Repository navigation
Commit e44dd47
authored
fix(server): keep a failed cursor key write from bricking the data location (#2061)
* fix(server): keep a failed cursor key write from bricking the data location
os.write may accept fewer bytes than requested. The previous code raised on a
short write but left the incomplete key file behind, and _read() then rejects
that file on every later start, so one failed write made the data location
permanently unusable with no way to regenerate the key.
Resume the write until every byte is on disk, and remove the key file when the
write cannot complete so the next start can create a new one. A write that
makes no progress is rejected instead of retried.
* fix(server): recover a cursor key left by an interrupted creation
A process killed between os.open and os.write leaves a key of the wrong
length on disk, and _read() raised ValueError for it. resolve_cursor_secret
runs from the server lifespan without a handler, so one crash made the data
location unstartable on every later start.
Treat a key of the wrong length as "no key yet": discard it and create a new
one. That is safe because the interrupted writer never returned the key to a
caller, so no cursor was signed with it. fsync the key before returning it so
a successful write also survives a power loss.
* fix(server): serialize cursor key recovery across starters
A starter that read a missing key could wake up after another starter had
published a complete one, delete it, and create a replacement. Both calls
succeeded with different keys, so cursors signed by the second starter no
longer verified against the key on disk.
Recovery (decide a key is debris, delete it, create a replacement) now
shares one cross-process lock with creation. A wrong-length file is only
treated as debris while holding that lock, where no other starter can be
midway through writing the key it is about to publish.
The lock file is kept after release: removing it would let a waiter on the
old inode and a newcomer on its replacement enter at the same time.
* fix(server): lock the cursor key file without seeding a marker byte
Two starters can both read the lock file as empty. If the first one pauses
after the size check and the second one writes its marker and locks byte 0,
the first one's write lands in a byte the second one has locked. Windows
rejects that write with errno 13, and the error escapes the retry loop, so
the first starter fails to start even though the persisted key is valid.
`_locking` accepts a range past the end of the file, so the marker is not
needed: locking byte 0 by itself keeps starters apart. The seeding step is
gone, and no write happens before the lock is held.
Checked with two real processes on Windows: a write into the held byte fails
with errno 13 while a second lock attempt is refused, and a zero-length file
locks and unlocks normally. The regression test enters the lock while another
process holds byte 0 and asserts the wait, so a lock that excludes nothing
cannot pass either; it fails at the old seeding line and passes now.
---------
Co-authored-by: SanJiu <tlyyxjz@users.noreply.github.com>1 parent bb9b075 commit e44dd47
2 files changed
Lines changed: 412 additions & 18 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
21 | 25 | | |
| 26 | + | |
22 | 27 | | |
23 | 28 | | |
24 | 29 | | |
| |||
27 | 32 | | |
28 | 33 | | |
29 | 34 | | |
| 35 | + | |
| 36 | + | |
30 | 37 | | |
31 | 38 | | |
32 | 39 | | |
| |||
46 | 53 | | |
47 | 54 | | |
48 | 55 | | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
53 | 100 | | |
54 | | - | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
55 | 117 | | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
61 | 146 | | |
62 | | - | |
63 | | - | |
64 | | - | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
65 | 166 | | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
66 | 170 | | |
67 | | - | |
68 | 171 | | |
69 | 172 | | |
70 | | - | |
71 | | - | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
72 | 197 | | |
73 | | - | |
| 198 | + | |
74 | 199 | | |
0 commit comments