Conversation
…thread
`issues/kernel/quiesce-runs-while-userland-still-does-io.md` recorded two
one-off T14 boots of `tests/metaldevicecase`. Every T14 boot of
`tests/metalcase` reproduces it: that config starts `sshd`, `sshd` spends a
second retrying a `netd` that has no NIC on this machine, and it is still
spinning in Ring 3 when the `reboot` job asks for the reset — so its own
`exit:` record lands after the boot's last word.
grep -n "Rebooting\.\|exit: sshd" <readback>/kernel.log (EXIT=0)
readback Rebooting. exit: sshd gap sshd cpu
metalcase 6.903 cpu4 7.033 cpu1 130 ms 2380 ms
run 34 6.101 cpu4 6.366 cpu1 265 ms 2382 ms
run 35 6.929 cpu4 7.035 cpu1 106 ms 2325 ms
On run 35 `sshd` burned 2,325 ms of CPU across the 2,326 ms between its spawn
at 4.709 and its exit, so it was never blocked: one CPU kept dispatching
userland across `Rebooting.`. All three came back with `stick_secs 0`, so the
device half stays answered and only the ordering half is open.
Which also names the asymmetry the fix has to live with: `bootlog::verdict`
reds on all three and `tests/metal-profile.toml` prices no number that would
notice.
`cargo test --lib`: 295 passed, 0 failed, EXIT=0.
`cargo test --workspace --exclude toyos-build`: 138 `test result: ok` lines, no
failures, EXIT=0.
The kernel half of this task is not here: the shared sysroot is claimed by
`toyos-aperture` for `host-bridge-abi`, so no kernel builds in this worktree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`quiesce` disarmed the watchdog, synced every filesystem and wrote the boot's
last word while every other CPU was still running userland. Both of those are
claims about a machine, and they were true only of the instant that made them:
on every T14 boot of `tests/metalcase`, `sshd` — spinning in Ring 3 retrying a
`netd` that has no NIC on that machine — wrote its own `exit:` record 106 to
265 ms after `Rebooting.`, and `bootlog::verdict` was EXIT=1 for it.
So the machine is stopped first, and the stop is taken at one place.
**The boundary.** `kernel_exit_to_user_check` is the one function every return
to Ring 3 goes through — the syscall gate, every device interrupt, the timer,
the TLB IPI, the trap epilogue and a task's first dispatch. A thread standing
there holds no kernel lock, has nothing in the block layer and nothing in
flight on any controller. Every record this kernel writes with a userland
author is written from inside a syscall, and a thread that never takes another
userland instruction never enters another syscall — so `quiesce::stop_here_if_due`
goes there, beside `exit_if_killed`, which stands there for the same reason.
**The band.** `toyos-sched` grows a third container beside `dying`: a `stopped`
band no pick serves and nothing is ever taken out of. A running thread enters
it through its own `dispose_stop` at that boundary; a thread *parked* inside a
syscall is marked where it lies by `stop_if_blocked`, whose CAS refuses a
running task precisely because a running one may hold a lock. `place` — the one
funnel every wake, adopt and retire goes through — routes anything carrying the
mark into the band ahead of the RT forward and ahead of the kill check, so a
deadline firing after the stop cannot put a thread back on a run queue to
finish its syscall and log. That last arm is what makes `Rebooting.` last by
construction rather than by luck, and it is sound because a thread that parked
holds no lock: `kernel/CLAUDE.md` already states that invariant.
**Tasks stop; CPUs do not.** Every CPU keeps `IF` set, keeps taking its LAPIC
timer and every device interrupt and keeps taking passes — it has no userland
left to dispatch. That is what the USB stop below the last word needs and what
`klogd` and `iod` need to carry the log to its volume. `deadline::this_cpu`'s
freeze-inside-a-pass is the wrong shape here: it would strand whatever lock the
thread on that CPU held, and `sync_all` is the first thing that would wait.
**The carve-out** is `logd`, and the kernel had no name for it. `wait_for_durable`
waits on a word any `Rights::LOG` holder publishes, so `log::user` now records
the pid of whoever *moved* that word and `Stage::ExceptLog` leaves that one
process runnable until the wait returns. Then `Stage::All`, then the reset. A
machine where nobody ever claimed durability carves nothing out and ends on
`wait_for_durable`'s existing two-second budget, with the record it already
writes.
**The record**, beside `irq:` and `nvme:` and above the last word:
stop: 9 of 9 userland thread(s) stopped across 2 cpu(s) in 0 ms over 14
sweep(s), 0 block operation(s) still open
The open-operation count is the block layer's own, taken inside `block::begin_operation`
rather than derived from the stop, so the two can disagree. The bound is
`quiesce::PARK` — one `QUANTUM_NS` plus one `block::OPERATION`, the two things a
thread that must stop can be doing — and its expiry is a clause in the same
record, never a hang.
Decisions in `toyos-quiesce/`: which thread must stop, when the stop is over,
and what the record says. Effects in `kernel/src/quiesce.rs`.
THE TWO CHECKS, both run:
* Negative control — the whole change reverted onto the base, with the test
kept: `quiesce_stops_the_machine` reds on lines reaching the console after
the boot's last word. Unit-level, `place`'s stop arm deleted and nothing
else: `a_wake_for_a_stopped_task_lands_in_the_band_and_not_the_run_queue`
reds `the wake reached the band: left 0, right 1` (EXIT=101).
* Independent oracle — the block layer's count, which the stop does not
produce; and the machine, whose readback is requested separately.
GATES
cargo test --lib 295 passed, EXIT=0
cargo test --workspace --exclude toyos-build EXIT=0
cargo test quiesce_stops_the_machine 1 passed, EXIT=0
cargo test in toyos-sched/ 72 passed, EXIT=0
`quiesce_stops_the_machine` and `quiesce_writers` are new registered names and
carry the `UNMEASURED` marker, which is provisional by declaration: the price
gate reds on this commit, CI measures them, and the next commit replaces the
markers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`quiesce_stops_the_machine` asserted that nothing follows `Rebooting.` and that
assertion was green on the base as well: the writer threads produce no kernel
record, and under QEMU nothing else was left to write one. A judge that cannot
fail is not a judge.
So each writer prints one line per pass. Six of them, with `quiesce-late-word`'s
staged hundred milliseconds of shutdown still to run, put lines under the boot's
last word on a machine that was not stopped — and on one that was stopped not
one of them can, because a stopped thread takes no further userland
instruction.
cargo test quiesce_stops_the_machine 1 passed, EXIT=0
[power] the machine stopped before it claimed anything:
[kernel 0.909 cpu0] stop: 9 of 9 userland thread(s) stopped across 2 cpu(s)
in 0 ms over 20 sweep(s), 0 block operation(s) still open
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Seen twice in one session on a host running ten worktrees: `cargo test <name>` panicked at `src/toolchain.rs:940` on a toolchain directory another worktree's exclusive `give the toyos toolchain its own cargo` phase was midway through writing. The refusal is right about the state and wrong about the cause, and a plain re-run passed both times with nothing changed. Filed, not fixed: it is the build system and not this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Rebooting. claims anything about it
Run 41 booted all four arms on 52c8a3f and the driver's verdict was EXIT=0 on every one, with the boot stick enumerated 0 s after each. `tests/metalcase` is the boot this branch exists for. Every earlier readback of it put `exit: sshd pid=8 code=0` 106 to 265 ms after the boot's last word; run 41's kernel log carries **one** `exit:` record in the whole boot — `netd`, at 4.405 s — and ends: [7.955 cpu4] stop: 7 of 7 userland thread(s) stopped across 8 cpu(s) in 129 ms over 67953 sweep(s), 0 block operation(s) still open [7.955 cpu4] usb-quiesce: disk 0 implements no SYNCHRONIZE CACHE, so it owed none [7.955 cpu4] Rebooting. `sshd` never exited at all: the stop caught it alive and it took no further userland instruction. `tests/metaldevicecase` — megabytes written and read back before the reset — stopped 5 of 5 in 0 ms over one sweep, `Rebooting.` last, stick at 0 s on the tightest ceiling in the file. `deadlinewedge` and `hardlockup` are unchanged and carry **no** `stop:` record, which is what their reset path predicts: both go through `acpi::reset_now` and neither calls `quiesce`. A record on either would have been a finding. Measured, into the rows that had no reading and only those: boot.metalcase.park_ms 129 (ceiling 2010) boot.metalcase.park_open_operations 0 (ceiling 0) boot.metaldevicecase.park_ms 0 boot.metaldevicecase.park_open_operations 0 boot.metaldevicecase.complete_ms 1255 boot.metaldevicecase.back_secs 51 boot.metaldevicecase.stick_secs 0 (ceiling 5) The ceilings do not move. `quiesce::PARK`'s derivation is the bound; 129 ms is one workload's one reading and the file's own rule is that `measured` is never the gate. `issues/kernel/quiesce-runs-while-userland-still-does-io.md` is deleted: the device half was already answered and the ordering half is now answered on the machine. Its one citation, in `issues/hardware/a-t14-boot-wedges-after-a-jobs-exit-and-nothing-said-why.md`, is rewritten rather than dropped — that issue's `hand_back` candidate is narrowed by the stop and not eliminated by it, and it now says so without pointing at a file that is gone. What run 41 found that this branch does not fix, filed at `issues/kernel/the-stop-sweeps-at-memory-speed-and-names-no-thread.md`: 67,953 sweeps in 129 ms is 1.9 us apiece, each taking the machine-wide process table lock, so the loop may be contending with the thread it is waiting for — and the record names a count and never a thread, so the log cannot say which of the seven took the time. cargo test --lib 295 passed, 0 failed, EXIT=0 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Japabu
marked this pull request as ready for review
September 14, 2026 07:03
…and its sweep has a cadence The review of PR #452 sent the branch back. This answers it. The carve-out was minted from `cursor.durable`, a value any `Rights::LOG` holder writes, so whichever program last moved the word decided which process survived a stop-the-world; five committed configs grant `logread` to two programs, and in `logrotatecase` the second holder also holds `power`. The identity now comes from where the capability was demanded — `sys_log_read` records its caller after `demand_syscap` succeeds — and the carve-out is every holder rather than the newest claimant, so an impostor can join it and never take it off `logd`. `Stage::ExceptLog` carries no pid; `Thread` carries the fact, and `DURABLE_PID` is gone. The sweep polled at memory speed: 67,953 sweeps in 129 ms on the T14, taking the machine-wide process table each time and contending with the very threads it waited for. It now waits one `QUANTUM_NS` between sweeps. The guest reads 2 sweeps where it read thousands, and the half of the tracked weakness that survives — the record names a count and never a thread — is narrowed to that. `Record`'s wire form is rendered and parsed in `toyos-quiesce` and nowhere else: `src/bootlog.rs`'s second spelling of `stop: ` is deleted, `src/metal.rs` reads a `Record` rather than splitting strings, and the harness's three constants are gone. A round-trip test fails on a reworded `Display` arm. The second stage discarded a `#[must_use] Record` and nothing tested it at all. It runs below the boot's last word, so its record goes on the black box, which is the one channel a reset's own account has; `power::done_chain` reads it back off the page after the seal, and deleting the call reds `blackbox_done_chain` on both the guest and the bench. `OPEN_OPERATIONS` counted operations `iod`, `usbd` and `xhci::flush_disks` begin, which the stop never stops, so its ceiling of 0 judged something the stop does not control; it now counts only the threads the stop stops. It was also asserted in its zero state alone: the record now carries how many such operations the boot began, and the guest test refuses a zero that is a counter which never counted (14,859 on that boot). The rest, each from the review: the metal suite could not see a stop that gave up, because an expiry spends its budget and reads under its own ceiling — `Readback::stop_completed` reads the shortfall clause for every boot; the headline judge passed vacuously on a boot that never wrote `Rebooting.`; the caller's `map_or(0, 0)` silently exempted whichever thread held those ids and is now a refusal by name; the kick-every-other-CPU loop was a third copy and is `apic::kick_all_but_self`, with `deadline.rs` and the panic drain on it; `stopped_len()` had no caller and is now the `sched:` census's third band; the actuator's doc named a test that does not exist. `toyos-sched/sim` did not compile against `Container::Stopped`, which no gate on this branch had run. `boot.metalcase.complete_ms` and `back_secs` were stale against run 41 and are its readings, 1260 and 45. The four `measured` fields on the park rows are dropped: they were read off a kernel whose record wording and sweep cadence this commit changes, and no machine has answered the new one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Four arms on the T14 at 22258c9, EXIT=0 on every one, the stick enumerated 0 s after each. `tests/metal-profile.toml` takes their readings. metalcase stop: 7 of 7 userland thread(s) stopped across 8 cpu(s) in 70 ms over 8 sweep(s), 0 of 9137 userland block operation(s) still open (kernel.log:330) metaldevicecase 5 of 5 ... in 0 ms over 1 sweep(s), 0 of 14192 ... (kernel.log:406) `Rebooting.` is the last line of both, two lines under the record. Neither wedge boot wrote one, which is what they owe: both reset through `acpi::reset_now` and `quiesce` is not on that path. Against run 41's pre-cadence kernel on the same config: 129 ms over 67,953 sweeps becomes 70 ms over 8. The sweep is no longer contending with the threads it waits for, and what is left is one thread's remaining syscall — which is the reading `issues/kernel/the-machine-stops-without-saying-which-thread-took-the-time.md` says the record cannot attribute to a thread, and still cannot. Every `measured` field these four boots produce is run 46's, not some of them: `measured` is the last reading the machine gave, and updating one boot and not its neighbours is the finding this branch was already sent back for. metalcase 1260 -> 1199 ms and 45 -> 51 s, metaldevicecase 51 -> 65 s, deadlinewedge 1199 -> 1254 ms, 236 -> 231 s and 61 -> 60 ms late, hardlockup 1201 -> 1206 ms and 172 -> 170 s. The four park rows are priced for the first time. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
… ran
The safe point checked the kill mark before the stop mark, against what
toyos-sched declares and what `stopping_outranks_killing` asserts: a thread
carrying both took `Dispose::Exit` and unwound into `process.rs`'s
`log!("exit: ...")` — the record this branch exists to keep out from under the
boot's last word.
`TaskShared::at_safe_point` is now that rank, declared beside `STOP` and `KILL`
in the scheduler core and read by `scheduler::leave_ring3_if_due`, which is one
call and one match at `kernel_exit_to_user_check`: `exit_if_killed` and
`stop_current` are gone, so the two sites that could disagree are one site with
no order in it. `stopping_outranks_killing_at_a_safe_point` reds on the
declaration. `object::ops`'s fsync retry loop asked the same question about the
kill mark alone and now asks this one, so a stopping thread stops retrying at
once instead of spending a block-layer deadman inside the park.
`quiesce_stops_the_machine` asserted nothing about its six writers: a boot whose
workload died left nothing under the last word either and passed every judge.
It now refuses a boot in which any writer wrote nothing *above* the last word,
by that writer's own line, before it reads what follows it.
`LOG_READERS` was appended to and never removed from; `teardown_bookkeeping`
now forgets an exiting pid, so the carve-out set is bounded by the live process
count and an exited pid holds no carve-out.
The sweep counted a thread between its table insert and its task mint in
neither band; it is counted running, so the record cannot say "N of N stopped"
with a thread still to be dispatched.
`Progress`'s three variants had one consumer reading one bit of them:
`Sweep::keep_waiting` replaces it. `Stage::must_stop` takes a `ThreadId` where
it took an untyped pair, and `Thread` carries one.
Prose deleted, not rewritten: block.rs's false claim about `iod`/`usbd` (the
xHCI flush runs on the caller's own userland thread and is counted),
`toyos-quiesce`'s "every process that can satisfy the wait" (the set is every
process that has *read*), `apic.rs`'s title line, quiesce.rs's copy of it,
`stop_current`'s fourth spelling of the boundary invariant, machine.rs's
appended narration, power.rs's four-bullet restatement of the four checks
beneath it and its copy of machine.rs's page-record contract,
quiesce_writers.rs's copy of that, tests/toyos.rs's transient UNMEASURED note,
and the sentence about "the two lateness fields" that this branch's two park
fields made false.
`idle_is_spinning`'s fixtures spell the `stopped=` field the kernel writes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`writer(s) [1, 2, 3, 4, 5] of this boot's 6` rather than five copies of the line prefix with the match's trailing space in each. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`object::ops`'s fsync retry loop asked the kill mark, and round two widened it to both marks. That widening is withdrawn: a thread inside a sync may be abandoned when it is killed, because its caller is gone, but not when the machine's stop names it — `quiesce` stops userland and then claims every filesystem is synced, and a sync the stop cut short is a claim about work that did not finish. Stage two runs below `sync_all`, so a `logd` fsync abandoned there had nothing left to flush it. The rank at the Ring 3 boundary is untouched: `leave_ring3_if_due` is still one call and one match over `TaskShared::at_safe_point`, which is what the review asked for. `driver::current_kill_pending` comes back for the one caller that wants the kill mark alone. Run 48 of the T14 is what prompted this, and it did not prompt it for the reason it looked like. The full account is `issues/diagnostics/a-t14-wedge-ran-the-deadline-out-and-sealed-nothing.md`: `tests/metalcase` wedged after `exit: netd` at 3.7 s, ran its 120 s boot deadline out, sealed no record, and was reset without a sync — which is what tore the volume the checker refused. The partition carries no `Syncing filesystems...`, no `stop:` and no `Rebooting.`, so the shutdown's stop never ran on that boot and the torn FAT is downstream of the wedge. The shape is the one `issues/hardware/a-t14-boot-wedges-after-a-jobs-exit-and-nothing-said-why.md` records, whose exit condition this occurrence still did not meet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
…y right-holder The first stage exempted every process that had ever called `SYS_LOG_READ` holding `Rights::LOG`. Ten committed configs grant `logread` to two programs, nine of them under `tests/` (`for f in $(git grep -l logread -- '*/system.toml' 'system.toml'); do grep -c 'syscap = \[.*logread' $f; done | grep -c '^2$'`), so on `tests/testcases` and `tests/metalcase` the carve-out was `logd` *and* `test-runner` — which writes files and spawns. The test runner therefore kept running across `sync_all`, which is the hazard stated in the comment directly above that line, and nothing synced after `Stage::All`. The wait is what names the carve-out. `log::wait_for_durable` polls `log::user::durable_ns`, and that word moves only where a caller's claim advances it, so the pid that advanced it is recorded beside it and the carve-out is that one process. A reader that has published nothing has ended nobody's wait and stops in the first stage with everything else; a machine where nobody has published carves nothing out and ends on the wait's existing two-second budget with the record it already writes. On every committed config the publisher is `/system/bin/logd` and nothing else. Two atomics and one relaxed store where the word actually moves, so the machine-wide `Lock<Vec<u32>>` is off `SYS_LOG_READ` — a syscall `logd` polls for the life of the boot — and the set that had to be pruned at process teardown goes with it. What the one carved-out process can still do between `sync_all` and the reset is `issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md`: it writes to an open file that no sync of the kernel's reaches, and a syscall it is already inside when the second stage begins runs to its end. That is what is left of the deleted `quiesce-runs-while-userland-still-does-io.md` once the general stop exists, and it is narrower than that file's claim in the one way that matters: no process the wait does not depend on runs at all. `quiesce::PARK` now says what it is. One syscall may open several block-layer operations in a row and `block::DEADMAN` is what bounds that sequence, so the budget was never the bound its derivation claimed. The fifteen `park_ms` rows go with the claim: a stop that gives up spends its budget and no more, so that ceiling could only be exceeded in the case `Readback::stop_completed` already reds on for every boot. `park_open_operations` stays — the block layer produces it, so it is the one number that can contradict the stop. Deleted prose: the citation refuted by `tests/metalcase/system.toml:57`, two tracker citations that resolve to nothing from a clean checkout, the argument for another module at `block.rs`'s statics, the restatement of `_deadline`, and the sentence a round-2 deletion left broken in `tests/common/metal.rs`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
… they are ready Four of six runs of `quiesce_stops_the_machine` at this head red in three modes, recorded in the branch's staging record for run 52. Two of the three were judges asserting what the design does not promise. None was the stop. **The open-operation count included the one process the stop deliberately leaves running.** `block::begin_operation` counted every operation opened by a userland thread, and the record reads that count at the end of stage one — where the carve-out, the process the durability wait depends on, is still running by construction. The `1 of 8466` recorded at this head is that process inside a device transfer: the machine working exactly as the stop intends, judged against a ceiling of 0. An operation exists only while its opener is executing inside the device. Every `begin_operation` is a driver leaf (`nvme`, `usb_storage`, `xhci`), its guard lives inside one `BlockDevice` call, and no disk wait in this kernel can park — so a thread the stop has stopped holds none, and that, rather than "userland holds none", is what the block layer can contradict the stop with. The count is now kept per stage: an operation is closed by the stage that stops the thread that opened it, decided at the open and carried on the guard so the two ends cannot disagree. `Stage::covers` is the one declaration and `must_stop` and `Stage::stopping` both read it. The ceiling of 0 does not move, because it is now the claim the design actually makes; stage two counts the carve-out's operations as well, and `done_chain` reds on a non-zero there, which is the only judge that sees the log writer's own operations closed. **The workload's readiness was a sleep.** `quiesce_writers` spawned six writers, slept 300 ms and asked for the reset, and the judge then required every writer's own line above the boot's last word. Under load a writer had not been scheduled inside that window, so the judge red on a precondition the guest never established. The writers now say so: each signals after its first completed write-and-fsync pass, and the reset is asked for once all six have. A writer that never reaches its loop is now one line naming how many did. The park's budget does not move. Twelve boots at this head measured it at 10-169 ms against `quiesce::PARK`'s 2,010 ms, on a quiet host and under 24 spinners alike, and the worst of them is the quiet host's. The readings and the reason a wider budget is not free — `block::DEADMAN` is 120 s and the boot deadline runs straight through `quiesce` — are in `issues/kernel/the-machine-stops-without-saying-which-thread-took-the-time.md`. Measured here, all at feffffb before this commit: five quiet-host runs green; four loaded-host runs, three red, every red the writers' precondition and every one of them carrying `9 of 9` stopped and `0` open. Gates, each with the exit status of the command itself: cargo test -p toyos-sched -p toyos-quiesce 73 + 11 passed EXIT=0 cargo run -- --build-only EXIT=0 cargo test --lib 295 passed, 1 ignored EXIT=0 cargo test --workspace --exclude toyos-build 140 ok lines, 0 failed EXIT=0 cargo run -- --clippy EXIT=0 cargo test --test toyos-build -- quiesce_stops_the_machine x3 EXIT=0 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
…itness goes The fifth review of PR #452 sent the branch back on a pid. This answers it. `DURABLE_PID` was minted from `cursor.durable`, and that pid decided which process survived a stop-the-world and ran across `Rebooting.`. Any holder of `Rights::LOG` writes that word, so a second `logread` holder calling `publish_durable` took the carve-out off `logd`; stage one then banded `logd` and the wait it exists for could not be satisfied. Ten committed configs grant `logread` to two programs. A pid a caller nominates for itself through a syscall is pid-as-authority, self-claimed and refused by no name, in a tree where a process holds exactly what its parent moved into it. **Why the pid came back, stated as the tension it was solving.** Round 1 struck it and `22258c9c` put the carve-out on the capability: every caller that passed `demand_syscap(_, Rights::LOG)`, kept in a `Lock<Vec<u32>>`. Round 2 refused that set — unbounded, sized from userland by a holder with `dup`, never pruned, and a machine-wide lock taken on every `SYS_LOG_READ`, a syscall `logd` polls for the life of the boot. Round 3 refused its width — on `tests/testcases` and `tests/metalcase` the second holder is `test-runner`, which writes files and spawns across the `sync_all` the stop is taken before. The pid answered both at once: one process, one word, no lock. It bought them with the caller's word, which is the one price this tree does not pay. Both are answerable without it. The identity is the capability again, so nothing a process says about itself joins it, and the two mechanical defects are answered where they were: `LOG_HOLDERS` is a fixed set of atomics, the shape `sched/kthread.rs`'s `ROWS` already uses for the same question about kernel threads — no allocation, no lock, readable with preemption off inside the block layer — and a surplus holder is refused by name and stops in the first stage like any other. `forget_log_holder` releases a slot at teardown, because a pid is issued again. The width is what is left: it is narrower than `main`, where every process runs across the sync, and wider than the wait, and it is recorded at `issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md` with its exit condition — a right of its own that `init` moves in, which is an ABI change and lands on its own pull request. The stage-two block count had no control that it ever counted. Deleting `+ OPEN_LOG_OPERATIONS.load(...)` left every judge green, because the second record is asserted only in its zero state and `begun` summed both buckets. It is deleted rather than given a judge, and with it `Stage::covers`, `Stage::stopping`, `open_count` and the guard's `stopped_by: Option<Stage>` — an abstraction feeding one uncontrolled assertion. One counter answers the same question: an operation is counted iff the stop's first stage stops the thread that opened it, decided at the open and carried on the guard as a bool. Every one-field mutation of what is left is now seen: the guard still decrements, so deleting the `fetch_add` underflows the count and the guest judge reds on a non-zero `in_flight`; deleting the counting altogether reds on `begun == 0`. A second `SYS_REBOOT`/`SYS_SHUTDOWN` caller overwrote the identity the first is exempt by, and the first can be parked in `block::between_attempts` inside the `sync_all` it is waiting on — so the second sweep banded the thread with the rest of the shutdown to perform, and that reset was never reached. The machine has one shutdown, claimed by name; the loser is refused, returns, and is banded at its own safe point like every other thread. The guest guaranteed six writer lines above the last word before asking for the reset, so the harness's silent-writer scan could not fire. One mechanism per premise: the guest establishes it, and the harness judges the kernel's own count, which the guest cannot write — a boot whose writers never ran stops a handful of threads and would otherwise pass every judge over a machine that had nothing to stop. `SPIN_UP` was 30 s against a host that waits 20 s, so the give-up line it exists to print could not be read; it is 5 s, inside that wait. Prose the review refused, deleted: the static's restatement of `Stage`'s contract at a block-layer static, `OpenOperation`'s narration of the two fields under it, `in_flight`'s argument for its own design, the writers' account of what the earlier implementation did, and the fourth spelling of the stage split in the harness. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Three files conflicted, and every hunk of both sides is accounted for.
`kernel/src/arch/syscall/machine.rs`: main added
`panic_console::log_census()` to the closing censuses and this branch added
`log!("{stopped}")` under them. Both survive, the census with its
neighbours and the stop record after all three.
`tests/common/metal.rs`: main added `Readback::panel` and the two priced
panel fields to the pricing loop; this branch added `park_open_operations`
to the same loop and replaced `field.ends_with("_lateness_ms")` with the
named `PATH_TAKEN` list. Both sets of fields are in the loop, and
`PATH_TAKEN` is an explicit three-name list rather than the widening
`8d5956f6` refused — the panel fields are not in it and are priced on every
boot, so a boot that reports none still reds. The comment above the loop is
this branch's trimmed one: the invariant it used to carry lives at
`PATH_TAKEN`'s own declaration.
`tests/metal-profile.toml`: both sides appended a block at the end, so both
blocks are kept, main's panel rows first. Four rows carry a `measured` both
sides moved — `boot.deadlinewedge.complete_ms`, `.back_secs`,
`.deadline_lateness_ms`, `boot.hardlockup.complete_ms` and `.back_secs`.
Main's are run 44's (`fc49eb5b`), this branch's are run 46's (`543a6329`),
and `measured` is the last reading the machine gave, so run 46's stand.
Main's run-44 `panel_max_us`/`panel_us` readings on the same two boots are
untouched by this branch and stay: those two boots now carry one run's
readings for the bounds and another's for the panel, which the next T14 run
over these boots resolves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`ci` run 34896137159 on pull request #452 measured `quiesce_stops_the_machine` at 4,434 ms across twelve shards — inside `FAST_COMMIT_MS` with margin, so the name stays `Tier::Fast`. That one row takes the run's price and only that row: replacing the whole merged artifact would re-price two hundred names this change does not own, which is what `47b794b9` was written about. The row also moves to where the merge writes it, after `query_pci_agreement`. `sys_shutdown` said "Does not return." and now can: a second caller of the shutdown is refused and gets `AlreadyExists` back. It says what `sys_reboot` beside it already said. `log::user`'s header said "No per-reader state", which `LOG_HOLDERS` is. The read still has none, and the header now names the table and says why it is in this file: `SYS_LOG_READ` is the one place in the kernel `Rights::LOG` is checked. `quiesce_writers`'s header said the harness matches `WRITING` by its spelling. It no longer does — the order judge reds on any line under the boot's last word, whoever wrote it. `WRITERS` is still spelt in both places. `usb_reset_hands_devices_back`'s deadline arm said it is armed because "the runner's loop can spawn another job into that gap". On `tests/jobdeadlinecase` the runner holds no `logread`, so this branch's first stage bands it before `sync_all` and that spawn cannot happen. The window is still real and still needs arming under QEMU: it is where the carved-out log writer is still putting bytes on the volume. And two the fifth review named that the tree still said: the chronology in `metal.rs`'s pricing loop, and `metal-profile.toml`'s restatement of the argument its own `ceiling_from` carries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Seven commits, up to dc38a05 (the ack-delay actuator measures the width of a shootdown's target set). No conflict: `git merge-tree --write-tree` was clean and the merge touched no file this branch changed by hand. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
…ling init nobody `quiesce::claim_the_shutdown` refused a second caller by name with nothing behind it: deleting it built and left every judge green, because no committed config issued two shutdowns. Now one does. `tests/quiescetwicecase` runs `quiesce_twice`, whose child closes a file and asks for the reset while the new `quiesce-drain-refuse` actuator keeps that file's flush owed — refused on `iod` every time, so the shutdown's own `writeback::drain_all` is what reaches it, and refused six times on the thread running the shutdown, so that thread parks in `block::between_attempts` for 10, 20, 40, 80 and 160 ms between its attempts. The job, a holder of the log capability and so still running in the stop's first stage, reads the kernel's log until the first of those refusals is in it and asks for the reset from the other CPU. The judge, `quiesce_refuses_a_second_shutdown`, reads where the kernel's refusal lands: once, between the first and the last of the actuator's lines, with one `Syncing filesystems...` on the boot, the second caller reporting `AlreadyExists` from Ring 3, and `Rebooting.` written by the first. With the guard deleted the second caller runs a second shutdown over the first — two syncs, no refusal — and the judge names that. `log::user`'s empty slot was `0`, which is `/system/bin/init`'s pid: the table issues from zero, init holds `Rights::LOG`, so `holds_the_log(0)` was false by construction and `note_log_holder(0)` a silent no-op. The empty slot is `u32::MAX`, the pid no table issues and the one `percpu` spells idle with, and both functions now refuse it by name. `quiesce`'s caller pair starts there too. `forget_log_holder` said a pid is issued again. It is not — `ProcessTable` is an `IdMap` and never reuses a key. The reason it exists is that the table has eight slots and every job a test boot spawns gets a `SysCap` dup, so the dead would fill it; that is what the site says now. `note_log_holder` loads before it exchanges: the steady state is `logd` finding itself in slot 0 on every read, which is a load and not a `lock cmpxchg` on the common `SYS_LOG_READ` path. `quiesce_stops_the_machine`'s workload judge asked for at least six threads and was met by three writers plus init, logd and test-runner. It asks for exactly nine — six writers plus those three — so a guest running three writers, or one that lost a writer to an I/O error before the reset, is the machine the record says it is not. A writer that fails says which call failed before it goes. The second stage's `in_flight` clause could not fire: the counter holds only operations of threads the first stage stops, and such a thread is inside a device with preemption off, which the stop counts as running first. Deleted. The check-then-act between `block::counted()` and a sibling thread's first `SYS_LOG_READ` is written at the site with its bound: a multi-threaded holder whose first read lands as the stop ends, which no committed config has. Prose: the count of programs holding the capability, the false "no process" sentence, the four spellings of the carve-out argument reduced to the one at `sys_log_read` where the capability is checked, the one-shutdown reason said once at `claim_the_shutdown`, the "cannot be linked" sentence said once beside `WRITERS`. The issue file's `metalcase` paragraph was false: `test-runner` holds the right there but runs no job list, never reads, and is not in the table — the carve-out on that boot is `logd` alone. `quiesce_refuses_a_second_shutdown` is registered with the `UNMEASURED` marker for fast CI to price. `quiesce-drain-refuse` has a `FLASHABLE` ruling of `Never`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`quiesce-drain-refuse` kept a closed file's flush owed by refusing it on `iod` too, and that put the entry in `iod`'s hands for the length of each attempt. `writeback::drain_retrying` counts the queue under the queue lock alone while `drain_one` pops under the VFS lock, so a shutdown whose count landed inside one of `iod`'s attempts counted zero, never parked, and the actuator fired no times: one boot in five read `refused the shutdown's drain 0 time(s)`, a red that was the judge's own premise failing and not the kernel's guard. Under the actuator `iod` now passes over the queue without popping, so the entry is never in its hands, and the refusals are eight where they were six: the thread running the shutdown is parked for 10 to 640 ms between attempts, 1270 ms in all, which is the window the second caller has to land in on a loaded host. Five boots in a row after the change read the refusal between the first and the last of the actuator's lines; the first of them has it on cpu1 one millisecond after refusal 1 of 8 on cpu0. The count-without-the-VFS-lock is `main`'s code and a defect in its own right: a shutdown can sync and reset while `iod` still owes a budget-refused flush. Filed with the boot that showed it and not fixed here: `issues/kernel/the-shutdowns-drain-counts-the-queue-while-iod-holds-an-entry.md`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
…ms it The previous commit kept `iod` off the write-back queue with a hunk of its own in `writeback::drain_one`. `main` has had that actuator all along: `writeback-stall` parks `iod` before it drains, which is exactly "a closed file's flush is the shutdown's own drain's to find". The hunk is deleted, the boot arms both names, and `quiesce-drain-refuse` is what it says again — eight refusals of the mirror write on the thread running the shutdown and nothing else. Five boots in a row read the kernel's refusal one line-group after refusal 1 of 8, as before. The judge counts the shutdowns first. With the guard deleted the second caller either bands the first where it is parked, or — arriving while the first is still running — interleaves with it, its stop giving up after 2019 ms on the one thread that will never reach a safe point and its `Syncing filesystems...` landing under the first caller's `Rebooting.`. Both shapes write that line twice, so both are now named as what they are: two shutdowns. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
…e checker says so
Round 7's reviewer named a hypothesis and the orchestrator ordered it run: a
thread `stop_if_blocked` marks while it is parked in `block::between_attempts`
never finishes its syscall, so whatever that syscall had half written stays
half written at the reset.
It is true. `quiesce_leaves_the_volume_whole` stages it with a third actuator
of `fat-mirror-write-refuse`'s shape, `quiesce-fsync-refuse`: from the moment
the machine is stopping it refuses the *active* FAT's write in nine attempts of
a `SYS_FSYNC` flush, after `set_fat_entry` has written the mirror. The caller
is `/system/bin/logd`; its ladder (parks of 10 ms doubling, 2550 ms over nine
refusals) outlasts `log::SHUTDOWN_DURABLE`, which a const assertion holds, so
the stop's second stage meets it parked over two FATs that disagree, with every
other userland thread already stopped. The second stage on purpose: `/log` has
512-byte clusters, so in the first stage `logd` itself would allocate the split
cluster within a few records and heal it by accident.
With the whole fix reverted (three files; M7-quiesce-8.diff) — BUILD EXIT=0,
TEST EXIT=1, the same failure both times, from `toyos-fat32-check`:
FAT 1 differs from FAT 0 at entry 44: 0x0FFFFFFF against 0x00000000
The fix is the unit the ruling names. `toyos_sched::task` carries `MID_UPDATE`
in the state word, so `stop_if_blocked` refuses a task parked inside an update
in the same exchange that reads `Blocked`; `block::begin_update` is the guard
and `ops::fsync` holds it across its ladder, the one retry ladder a userland
thread parks in. The sweep counts such a thread as running, so the stop waits
for it inside `PARK` and the record's shortfall clause speaks if it cannot.
With it — BUILD EXIT=0, TEST EXIT=0, the checker silent, and the kernel's own
lines:
[kernel 2.438 cpu0] shutdown: /log did not answer in 2000ms ...
[kernel 3.006 cpu0] fsync: /log/2026-09-18-151803.log durable on attempt 10 after 2567ms
The judge asserts the harm first, then that the arm fired nine times, that the
shutdown gave up inside that ladder and that the flush closed after it — a
whole volume that does not say so was healed by something else. The price is
that `fsync:` line, a kernel record under the last word on that boot;
issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md
carries it.
From the same review:
* `forget_log_holder` and its call are deleted. Its reason was false of this
kernel (a dup records nothing; a holder is recorded at its first read) and it
had no judge. A slot is never given back; no pid is issued twice.
* The one-shutdown claim moves to `kernel/src/quiesce/claim.rs`, which
`kernel-loom` compiles: `shutdown_claim` EXIT=0, and EXIT=101 under
`--features shutdown-claim-split` (a load, then a store) with `two callers
ran the shutdown`. `quiesce_twice`'s second caller now asks through
`SYS_SHUTDOWN`, so the one boot judges the claim on both power syscalls:
the claim moved into `sys_reboot` alone — BUILD EXIT=0, TEST EXIT=1, `this
boot ran 2 shutdowns`, both times; the guard deleted whole, the same. The
judge now counts the shutdowns before it asks how the machine ended, since a
second caller let in through `SYS_SHUTDOWN` may be the one that ends it.
* `Refused::ShutdownDrain.of` and the unused `Clone, Copy` go.
* The `counted()` check-then-act is
issues/kernel/the-stops-in-flight-count-decides-who-is-counted-at-the-open.md,
with the true bound: `test-runner` is a multi-threaded holder on
`tests/testcases`; what no config arranges is its first read beside a
sibling's operation as the stop ends.
* The workload judge says what its three are: `init` and `test-runner`'s main
and deadline threads; `logd` is carved out uncounted.
* Prose by deletion: the 1270 ms sentence, "the pid no table issues", and the
two-callers scenario, which the judge's doc now tells alone.
Also re-measured on this tree: the whole change reverted onto origin/main's
kernel and scheduler (21 files, judges touched 0) — BUILD EXIT=0, TEST
`quiesce_stops_the_machine` EXIT=1, `5 line(s) reached the console after the
boot's last word`; the holder call never taken — BUILD EXIT=0, TEST EXIT=1,
`/log did not answer in 2000ms` under the last word. After the last restore:
BUILD EXIT=0 and all three judges EXIT=0.
Found by reading and filed, not fixed:
issues/filesystem/a-refused-link-write-leaks-the-cluster-append-cluster-just-claimed.md.
`quiesce_refuses_a_second_shutdown` is priced: CI run 35355556050 at 4057307
finished (every guest shard green; `durations` red on the marker, as it must
be), and its `test-durations-merged` artifact reads 4249 ms across twelve
shards. That one row is taken and no other; 4249 is inside `FAST_COMMIT_MS`, so
the name stays Fast and the "Registered UNMEASURED" sentence goes with the
marker. `quiesce_leaves_the_volume_whole` is registered UNMEASURED in its turn.
Gates on this tree, 2026-09-21: the harness's Fast tier 346 passed, 1 failed,
347 total, 412.3 s, EXIT=1 — `handle_kill_policy`, `16 more killed processes
left more live objects behind: [("SharedMem", 5, 6)]`, green alone, on a host
another agent's work had come back to (the same tier took 147.3 s at 4057307);
`--known-red` answers NOT KNOWN-RED, the redlist holding that sentence as a
sighting retired 2026-09-04. `cargo test --lib` EXIT=101 on
`redlist::tests::every_row_can_say_what_it_claims` alone: four of main's rows
measured 2026-08-20/21 are now more than 31 days old; this branch does not
touch `src/redlist.rs`. Workspace host suites EXIT=0, five clippy invocations
EXIT=0 each. The loom control is wired into `.github/workflows/host-tests.yml`,
which `every_model_control_is_wired_into_host_tests` demanded by name.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
quiescedisarmed the watchdog, synced every filesystem and wrote the boot'slast word while every other CPU was still running userland. Both of those are
claims about a machine and they were true only of the instant that made them.
On every T14 readback of
tests/metalcasethis repository holds,sshd—spinning in Ring 3 retrying a
netdthat has no NIC on that machine — wrote itsown
exit:record afterRebooting.:Rebooting.exit: sshdstick_secsbootlog::verdictis EXIT=1 for each.issues/kernel/quiesce-runs-while-userland-still-does-io.mddoes not survive asit was written — no process without the log capability runs across the sync any
more — but it is narrowed, not deleted:
issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.mdiswhat still stands once the general stop exists, under a slug that states it.
The boundary
kernel_exit_to_user_checkis the one function every return to Ring 3 goesthrough — the syscall gate, every device interrupt, the timer, the TLB IPI, the
trap epilogue and a task's first dispatch. A thread standing there holds no
kernel lock, has nothing in the block layer and nothing in flight on any
controller. Every record this kernel writes with a userland author is written
from inside a syscall, and a thread that never takes another userland
instruction never enters another syscall — so the stop is taken there, at the
same boundary a killed thread's last exit is taken at.
One rank, applied at two sites, with a test on each. A thread can stand at
that boundary carrying both marks.
TaskShared::at_safe_pointis thedeclaration — written beside
STOPandKILLthemselves — andscheduler::leave_ring3_if_dueis one call and one match over it.CpuSched::placeapplies the same rank to a task that is not running and readsthe two words itself, because the RT forward sits between the two arms. Both
are covered:
stopping_outranks_killing_at_a_safe_pointreds on thedeclaration,
stopping_outranks_killingonplace's own order.Tasks stop; CPUs do not
Every CPU keeps
IFset, keeps taking its LAPIC timer and every deviceinterrupt and keeps taking passes — it has no userland left to dispatch. That is
what the USB stop below the last word needs and what
klogdandiodneed tocarry the log to its volume.
The carve-out is a capability, and the pid that was here is gone
wait_for_durablewaits on one word,log::user::durable_ns, and that wordmoves in exactly one place: a caller's clamped claim inside
publish_durable,reached only through
SYS_LOG_READ. An earlier head took the carve-out's namefrom there too — the pid that last advanced the word — and that was a defect.
A pid a caller nominates for itself through a syscall word is pid-as-authority,
self-claimed and refused by no name. Any holder of
Rights::LOGwritescursor.durable, so a secondlogreadholder could take the carve-out offlogd; stage one would then bandlogdand the wait that carve-out exists forcould not be satisfied.
The tension that put it there, stated rather than rediscovered. The first
implementation carved out the capability — every caller that passed
demand_syscap(_, Rights::LOG)— and it was refused twice, on two differentgrounds. Mechanically: the set was a
Lock<Vec<u32>>, unbounded, sized fromuserland by a holder with
dup, never pruned, behind a machine-wide lock takenon every
SYS_LOG_READ— a syscalllogdpolls for the life of the boot. Bywidth: ten committed configs grant
logreadto two programsand a holder joins the carve-out at its first
SYS_LOG_READ. Ontests/testcasesthe second holder istest-runner, which reads inside itslog-gateandlog-closebuiltins and writes files and spawns across thesync_allthe stop is taken before. Ontests/metalcasetest-runnerholdsthe right and its committed config gives it no job list — the image staged for
the T14 gives it the
rebootbuiltin alone, which reads nothing — so it neverreads and stops in the first stage; the carve-out there is
logdalone. The pid answered both objections at once —one process, one word, no lock — and paid for it with the caller's word.
Both are answerable without paying that. The identity is the capability again,
so nothing a process says about itself joins it. The mechanical objections are
answered where they were made:
LOG_HOLDERSis a fixed array of atomics — theshape
kernel/src/sched/kthread.rs'sROWSalready uses to answer the samekind of question about kernel threads — so there is no allocation, no lock, and
it is readable with preemption off inside the block layer; and a surplus holder
is refused by name and stops in the first stage like any other. A slot is never
given back: no pid is issued twice, so a dead holder's slot names nothing that
runs. An earlier head released slots at teardown, for a reason that was false of
this kernel — a dup records nothing, a holder is recorded at its first read, and
the tree's readers are
logd,console,test-runnerandquiesce_twice—and with no judge (the round-7 reviewer compiled the call out and the whole Fast
tier stayed green for it); the function and its call are deleted.
SYS_LOG_READtakes no lock of the carve-out's, as onmain, and in the steadystate —
logdfinding itself in slot 0 — it is one relaxed load and noread-modify-write.
An empty slot is
u32::MAX, not zero. Zero is/system/bin/init's pid: theprocess table issues from zero and init holds
Rights::LOG, so with zero as thesentinel
holds_the_log(0)was false by construction andnote_log_holder(0)asilent no-op.
u32::MAXis the pidpercpucannot represent, because it spellsidle with it; both functions refuse it by name, and
quiesce's caller pairstarts there too.
The width is what is left, and it is tracked rather than claimed away. It is
narrower than
main, where every process runs across the sync, and wider thanthe wait, which depends on one. The exit condition is in
issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md: aright of its own, moved in by
initfrom asystem.tomlrow naming the programthat writes
/log, so the kernel still reads a capability and not a claim. Thatis an ABI change and lands on its own pull request.
One check-then-act, tracked.
block::counted()reads holder status when anoperation opens, and a process becomes a holder at its first
SYS_LOG_READfromany thread. A sibling's first read while this thread is inside a counted
operation at the instant the first stage's record is taken leaves
in_flight = 1on anN of Nrecord — a false red, never a false green. A multi-threadedholder is not hypothetical:
test-runnerontests/testcasesis one. What nocommitted config arranges is its first read landing beside a sibling's
operation as the stop ends.
issues/kernel/the-stops-in-flight-count-decides-who-is-counted-at-the-open.mdcarries it with its exit condition.
One shutdown, claimed by name, and judged
CALLER_PID/CALLER_TIDare one global pair, and a secondSYS_REBOOTorSYS_SHUTDOWNcaller overwrote the identity the first is exempt by. The firstcaller can be parked —
block::between_attemptsparks inside the retry loopwriteback::drain_allruns — so a second caller's sweep bands the thread withthe rest of the shutdown to perform.
quiesce::claim_the_shutdownis oneexchange (
kernel/src/quiesce/claim.rs) taken before the watchdog is disarmed;the loser is refused by name and gets
AlreadyExistsback through Ring 3.That guard had no judge, and now has one.
tests/quiescetwicecaserunsquiesce_twice. Its child closes a 64 KiB file on/logand asks for thereset.
writeback-stallparksiod, so that file's flush is the shutdown's owndrain's to find; the new
quiesce-drain-refuseactuator refuses its FAT-1mirror write eight times as a budget expiry on the thread running the shutdown,
so that thread parks in
block::between_attemptsfor 10, 20, 40, 80, 160, 320and 640 ms between its attempts. The job itself — a holder of the log
capability, so still running in the stop's first stage — reads the kernel's log
until the first refusal is in it and asks again from the other CPU, through
SYS_SHUTDOWN: the first seat isSYS_REBOOT, so the one boot judges the claimon both power syscalls.
quiesce_refuses_a_second_shutdownjudges where the kernel's refusal lands:It asserts, in order: one
Syncing filesystems...on the boot; eight actuatorrefusals, the kernel's own count; the refusal by name once, between the first
and the last of them; the second caller's
AlreadyExistsreaching its console;Rebooting.; and QEMU'sguest-reset.With the guard deleted it is red, in two shapes, and neither is "no
Rebooting.". The prediction was that the banded first caller leaves themachine with no last word. What the machine does instead is run a second
shutdown — on every one of the sixteen further boots that kernel was given at
40573079(eight runs, each booted beside and then alone, EXIT=1 on alleight; the round-7 reviewer added twenty, 38 of 38 with the handoff's). In ten
of the sixteen the second caller finds the first parked, bands it, and performs
the whole shutdown itself: two
Syncing filesystems...on two CPUs — 1 ms aparton eight of the ten, 2 ms and 11 ms on the other two — one
stop:record andone
Rebooting., both the second caller's. In six it finds the first stillrunning between attempts and the two interleave:
— the second caller's stop spends its whole budget on the one thread that will
never reach a safe point, and then writes a sync line and a second last word
under the boot's last word. Both shapes write the sync line twice, which is
why that count is the judge's first assertion.
The stop leaves no filesystem update half made, and the volume checker says so
stop_if_blockedmarks any parked thread, andblock::between_attemptsparks athread between two attempts of one filesystem update. A banded thread never
finishes its syscall, so what a refused attempt left on the volume stays there
at the reset. The round-7 reviewer named that as a hypothesis;
quiesce_leaves_the_volume_wholeis the run.quiesce-fsync-refuse, a third actuator offat-mirror-write-refuse's shape,refuses the active FAT's write in nine attempts of a
SYS_FSYNCflush fromthe moment the machine is stopping — after
set_fat_entryhas written themirror, which is the order that function keeps. The caller is
logd. Itsladder's parks (10 ms doubling, 2550 ms over nine refusals) outlast
log::SHUTDOWN_DURABLE, and a const assertion holds that; so the shutdown givesup waiting for
/logand the stop's second stage meetslogdparked overtwo FATs that disagree. The second stage and not the first, on purpose:
/loghas 512-byte clusters, so in the first stage
logditself allocates the splitcluster within a few records and heals it by accident; below the second stage
nothing in the machine allocates again.
On the kernel without the fix — BUILD EXIT=0, TEST EXIT=1, the same
failure both times, from
toyos-fat32-check:The fix is the unit. The count the stop already keeps is per block
operation; what may not be cut is the filesystem's update.
toyos_sched::taskcarries
MID_UPDATEin the task's state word, sostop_if_blockedrefuses atask parked inside an update in the same exchange that reads
Blocked— aflag beside the word would be a read and a mark that can come apart.
block::begin_updateis the guard andops::fsyncholds it across its ladder,the one retry ladder a userland thread can be parked in. The sweep counts such a
thread as running, so the stop waits for it inside
PARK, bands it at the Ring 3return of that same
fsync, and says so in the record's shortfall clause if theupdate outlasts the budget. With it — BUILD EXIT=0, TEST EXIT=0:
The judge asserts the harm first (the checker, over a partition asserted clean
before the boot), then that the arm fired nine times, that the shutdown gave up
inside that ladder, and that the flush closed after it — a whole volume that
does not say so was healed by something else, and is a red.
What it costs, said rather than hidden: that
fsync:line is a kernelrecord under the boot's last word, on a boot whose shutdown has already said its
last lines are on the console only.
issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.mdcarries it as what is still owed.
Read on the way, filed, and not fixed — it is
main's and unmeasured:toyos-fat32'sappend_clusterclaims the new cluster and then links it, and arefusal of the link leaves that cluster allocated where
rollback_to— whichwalks the chain from the file's head — never reaches it, so the caller's retry
allocates another. That is a leak a successful retry does not heal, and it is
the sentence
log_flush_retryandfat_backing_revokedsay inissues/build/a-loaded-suite-reds-a-volume-checker-on-both-arms.md;issues/filesystem/a-refused-link-write-leaks-the-cluster-append-cluster-just-claimed.mdnames the host test that would show it. It is also why the actuator's first
refusal in an attempt is always the allocation's active-FAT write, which ends
the attempt: that is the one split
set_fat_entry's order makes a retry heal.The record, and the one counter behind it
toyos-quiescerenders it and parses it and nothing else spells it: the kernelwrites it,
src/metal.rsreads aRecordoff a stick andtests/common/power.rsreads one off a console.every_record_reads_back_as_itself_off_a_line_of_logis what a rewordedDisplayarm fails.The stage split is gone. An earlier head kept two open-operation counters so
the second stage's record could carry the carve-out's own operations — and that
second term had no control that it ever counted. It is deleted rather than given
a judge, and
Stage::covers,Stage::stopping,open_countand the guard'sstopped_by: Option<Stage>went with it. One counter answers the question: anoperation is counted iff the stop's first stage stops the thread that opened it,
decided at the open and carried on the guard as a bool. The second stage's
record prints the same word and the harness no longer asserts on it there: a
thread holding such an operation is inside a device with preemption off, which
the sweep counts as running, so that clause could never have fired before the
one above it.
The ceiling of 0 is a claim about a stopped machine, and it holds because no
open operation can outlive its opener's stop.
Lock::lockdisables preemptionand the guard re-enables it on drop, and every
begin_operationin the NVMe andUSB-storage paths sits inside a
BlockDevicemethod reached only throughPartition::read_blocks/write_blocks/flush— under that guard, on theopener's own thread, with no park anywhere in those wait paths. The one
begin_operationthat is not inside aBlockDevicecall is inkernel/src/drivers/xhci/mod.rs'sflush_disks: it runs on the reboot caller'sown thread and closes before either record is read, so it is outside both counts
by the time they are taken.
The two checks this change is judged by
Every mutation's build exit is given before its test exit — a mutation is a
measurement only once the mutated tree is shown to build.
BUILDiscargo test --test toyos-build -- --jobs 1 process_stats, which builds thetoolchain, the kernel, every guest binary and the image and then boots one
unrelated test;
TESTis the same runner on the judge named. Measured onthis head's tree, each restored from the index under a trap with the working
tree read back equal to it: the whole-change control, the whole of the
update-unit fix reverted, the holder call never taken, the guard deleted, and
the guard moved into one syscall. Measured at
40573079by the implementerand again by the round-7 reviewer, and not re-run here: the two one-field
mutations of the block layer's count and the three-writer guest; nothing this
head changes is on their path.
The negative control: the whole change reverted onto the base, judges kept.
git restore --source=origin/main --worktree -- kernel/ toyos-sched/, withorigin/mainat9f91b581, puts the base's mechanism — 21 files — under thisbranch's instruments; every judge, guest binary, boot config and profile row
stays the branch's (0 files differing under
tests src toyos-quiesce issues).kernel/src/quiesce.rssurvives as a file the base'smain.rsdeclares nomodule for, so it is not compiled. BUILD EXIT=0, TEST
quiesce_stops_the_machineEXIT=1, red on the defect itself both times:The update-unit fix, reverted whole —
MID_UPDATEout of the state word,block::OpenUpdategone,ops::fsyncholding nothing, three files — BUILDEXIT=0, TEST
quiesce_leaves_the_volume_wholeEXIT=1, the same failureboth times, quoted in the section above. Its independent oracle is
toyos-fat32-check, written from Microsoft's fatgen103 and sharing no code withthe driver it judges.
The independent oracle is the block layer's own count, which the stop does
not produce: it is incremented where an operation opens, in a driver leaf, and
the stop is judged against it. Both of its one-field mutations are seen.
Deleting the
fetch_addalone, the guard still decrementing — BUILD EXIT=0,TEST EXIT=1:
Making both ends agree that nothing is counted
(
counted() && core::hint::black_box(false)) — BUILD EXIT=0, TESTEXIT=1:
The carve-out is load-bearing, and its removal is seen too — in the form that
builds. Deleting the one call that records a holder does not build: the kernel
denies
note_log_holderandREFUSED_A_HOLDERas dead code, so that mutationmeasures nothing. The call compiled and never taken
(
if core::hint::black_box(false) { note_log_holder(..) }) — the capabilitystill checked, no holder ever recorded — bands
logdin the first stage, so thedurability wait can never be satisfied. BUILD EXIT=0, TEST EXIT=1, the
same failure both times:
The one-shutdown guard, whole and in both partial forms. The call, the
refusal,
claim_the_shutdown, its static and its module deleted — BUILDEXIT=0, TEST
quiesce_refuses_a_second_shutdownEXIT=1, the samefailure both times:
The claim moved out of
quiesce()intosys_rebootalone used to pass,because both callers used
SYS_REBOOT. The second caller now asks throughSYS_SHUTDOWN, and that mutation is BUILD EXIT=0, TEST EXIT=1, the samesentence both times. The judge counts the shutdowns before it asks how the
machine ended: the first run of that mutation read
QEMU stopped this guest for "guest-shutdown"alone on one of its two boots, which is the second callerwinning — true, and not the harm.
A load and then a store in place of the one exchange is reached by no boot:
the second caller arrives a millisecond after the first. It is
kernel-loom/tests/shutdown_claim.rs's, overkernel/src/quiesce/claim.rsitself and not a transliteration: EXIT=0 as written, EXIT=101 under
--features shutdown-claim-splitwithtwo callers ran the shutdown, wiredinto
host-tests.ymlbeside the other model controls.The workload judge fires on the partial fix that used to pass. It asked for
at least six threads and was met by three writers plus three others. It now asks
for exactly nine: six writers,
init, andtest-runner's main and deadlinethreads.
logdholds the log and the first stage's sweep passes over ituncounted; the job's own main thread is the caller. Measured at
40573079, bythe implementer and again by the round-7 reviewer — the guest run with three
writers where the harness is told six, BUILD EXIT=0, TEST EXIT=1,
stop named 6 ... make 9, the same failure both times. This head changes thatmessage's words and not its count.
The green arm was re-measured after the last restore: BUILD EXIT=0, all
three judges EXIT=0, with the record, the refusal and the closed flush
quoted above.
The second oracle is the machine: a T14 readback with
Rebooting.last and thestick enumerating 0 s after, which is owed and named below.
The guest says when it is ready; the harness counts what the kernel stopped
quiesce_writersputs six threads into a write-and-fsync loop and asks for thereset only once every one of them has completed a pass. The harness judges the
kernel's own thread count, which the guest cannot write, and judges it exactly:
a writer whose loop ends on an I/O error now says which call failed, and leaves
eight where nine are asked for.
Gates, each with the exit status of the command itself
All on this head's tree, 2026-09-21.
cargo test --test toyos-build(the harness's whole Fast tier)cargo test --libcargo test --workspace --exclude toyos-buildtest result: oklines, 0 failed, EXIT=0toyos-schedalonea_task_parked_mid_update_refuses_the_parked_markamong them, EXIT=0kernel-loomshutdown_claim/ the same under--features shutdown-claim-splithostjob's fivecargo clippyinvocations, spelled assrc/clippy.rsspells themcargo test --lib's red ismain's, and it is a date.redlist::tests::every_row_can_say_what_it_claimsrefuses four rows measured on2026-08-20 and 2026-08-21 as
more than 31 days ago, and still standing(
console_line_atomicity,kill_while_blockedtwice,xhci_full_speed_device).This branch does not touch
src/redlist.rs(git diff origin/main -- src/redlist.rsis empty), soorigin/mainat9f91b581reds the same waytoday, and a plain
cargo teststops at that target before it boots a guest —which is why the tier above is run by its own target. The rows are their
owners' to re-take, retire or delete; nothing here ages them away.
The tier's one red is not shown to be this diff's, and it is not on the
list.
handle_kill_policy:16 more killed processes left more live objects behind: [("SharedMem", 5, 6)], thenALONE handle_kill_policy: GREEN, in atier that took 412.3 s against the 147.3 s the same tier took at
40573079—another agent's work resumed on this host beside it, and the same boot shows
usb-storage: ... no answer in the data phase in 2000 ms.cargo run -- --known-red handle_kill_policyanswers NOT KNOWN-RED: the redlist carriesthis sentence (with
Processwhere this one saysSharedMem) as a loaded-hostsighting retired on 2026-09-04 by six green tiers. It is a machine-wide census
either side of a kill on a shared boot; nothing this branch changes allocates or
frees a
SharedMem. One sighting prices nothing, and it is the redlist's tore-open at its owner, not re-run away here.
fs_dirs_durable, the tier's red at40573079, was green in this run.Two reds seen at earlier heads, named rather than re-run away.
blocked_dump— the round-6 reviewer's tier run atc2352819readPANIC: ... src/sched/dump.rs:134:5: the blocked-task dump ran under a lock: preempt depth 2, backtracedump::request <- driver::drain_irqs <- driver::pass_block <- completion::wait_inner <- inbox::submit, green alone. It did not occur inthis head's tier run. The path is
main's: this branch's only change underkernel/src/sched/is 14 lines indriver.rs, andcompletion.rs,dump.rsandinbox.rsare untouched. It is a kernel defect on a path thisbranch does not own and belongs to its owner in the redlist.
guest (11), CI run34896137159at8fdb28b9:FAIL metal_device_probe: usbwrite: the job refused — IoFailed, green alone. The diff does touch theguard every USB write opens (
block::begin_operation) and the placement everywake crosses (
CpuSched::place), so it was measured rather than read: asame-session A/B on
metal_device_probe, three boots at40573079and threewith the working tree set to
origin/mainatdc38a054(0 files differingfrom it) — EXIT=0 on all three at head, EXIT=0 on all three on main.
CI's own run at
c2352819(35100889732) concludedsuccesswith that shardin it. The round-6 reviewer found the line in one of the last 40 failed
ci.ymlruns and in no other; here it reproduced on neither arm. Six boots donot price a rate that small, so this says the diff is not shown to cause it,
not that it cannot; it is the redlist's to carry at its owner.
The merge, and the numbers it had to choose between
origin/mainatdc38a054is merged in, 7 commits past the previous mergebase, with no conflict. The earlier merge of 45 commits at
8261079ehad threeconflicted files, every hunk of both sides accounted for in that merge commit's
message, and one choice that still stands:
boot.deadlinewedge.complete_ms,.back_secs,.deadline_lateness_ms,boot.hardlockup.complete_msand.back_secscarry ameasuredboth sides moved — main's from T14 run 44, thisbranch's from run 46.
measuredis the last reading the machine gave, so run46's stand; main's run-44
panel_max_us/panel_uson those same two boots areuntouched by this branch and stay. The run-56 staging below supersedes all of
them at once.
The machine
The four T14 boots are owed at this head, and the arms are the
orchestrator's. The run-55 images were staged at
40573079and are stale: thishead changes the kernel every one of them carries (
block.rs,object/ops.rs,quiesce.rsand its newclaim.rs,log/user.rs,process.rs,fat32_adapter.rs) and the scheduler's state word. A run-55 reading stilldescribes this head in one respect only — the first stage's record on a boot
where nothing is parked in an
fsyncladder takes the same path — and it doesnot describe the second stage, which now waits such a ladder out. They are
restaged at
target/metal-quiesce-run56/from a tree clean before and after atthis head, with hashes, invocations and expected lines in
t14-request-quiesce-6.txt. Each of the four carries the stop record's literalonce; an image from a worktree without this branch carries it zero times and
Rebooting.once, which is what says the reader works.What this costs, declared
tests/metal-profile.tomlgains 15 rows —park_open_operationsfor each ofthe 15 boots that reset through
quiesce.park_msis priced nowhere and thatis the point: a stop that gives up spends its budget and no more, so a ceiling
on the duration could be exceeded only in the expiry case, which
Readback::stop_completedalready reds on for every boot the suite reads backby reading the record's own shortfall clause.
Three registered names.
quiesce_stops_the_machineis priced: CI run34896137159measured it at 4,434 ms across twelve shards, insideFAST_COMMIT_MS, so it staysTier::Fast.quiesce_refuses_a_second_shutdownis priced by CI run
35355556050at40573079, whosetest-durations-mergedartifact reads 4,249 ms across twelve shards; that one row is taken from it
and no other, and it stays
Tier::Fast.quiesce_leaves_the_volume_wholeisnew and carries the
UNMEASUREDmarker, so this landing pays the two CI cyclesa new name costs a second time; the dev host ran it in 4 s, most of it the
2,000 ms the shutdown waits for a
/logthe actuator keeps from answering.quiesce_writersandquiesce_twiceare guest binaries onRUST_SKIPandcarry no row. Two new actuators,
quiesce-drain-refuseandquiesce-fsync-refuse, each with aFLASHABLEruling ofNever, and onekernel-loomcontrol,shutdown-claim-split.quiesce::PARKsays what it is —a budget, not a bound the kernel can prove — and its expiry is a clause in the
record rather than a hang.
Decisions in
toyos-quiesce/(pure, 10 tests); effects inkernel/src/quiesce.rs.Filed and not fixed:
issues/build/two-worktrees-race-provisioning-the-toolchains-cargo.md,issues/kernel/the-machine-stops-without-saying-which-thread-took-the-time.md,issues/diagnostics/a-t14-wedge-ran-the-deadline-out-and-sealed-nothing.md,issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md,issues/kernel/the-stops-in-flight-count-decides-who-is-counted-at-the-open.md,and — in
main's own code —issues/kernel/the-shutdowns-drain-counts-the-queue-while-iod-holds-an-entry.md(
writeback::drain_retryingcounts the queue without the VFS lockdrain_onepops under, so a shutdown can sync and reset while
iodstill owes abudget-refused flush) and
issues/filesystem/a-refused-link-write-leaks-the-cluster-append-cluster-just-claimed.md.🤖 Generated with Claude Code
https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK