Skip to content

The machine stops before Rebooting. claims anything about it - #452

Open
Japabu wants to merge 22 commits into
mainfrom
wt/toyos-quiesce
Open

Japabu wants to merge 22 commits into
mainfrom
wt/toyos-quiesce

Conversation

@Japabu

@Japabu Japabu commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

quiesce disarmed the watchdog, synced every filesystem and wrote the boot's
last word while every other CPU was still running userland. Both of those are
claims about a machine and they were true only of the instant that made them.
On every T14 readback of tests/metalcase this repository holds, sshd
spinning in Ring 3 retrying a netd that has no NIC on that machine — wrote its
own exit: record after Rebooting.:

readback Rebooting. exit: sshd gap stick_secs
metalcase (2026-09-07) 6.903 cpu4 7.033 cpu1 130 ms 0
run 34 6.101 cpu4 6.366 cpu1 265 ms 0
run 35 6.929 cpu4 7.035 cpu1 106 ms 0

bootlog::verdict is EXIT=1 for each.
issues/kernel/quiesce-runs-while-userland-still-does-io.md does not survive as
it was written — no process without the log capability runs across the sync any
more — but it is narrowed, not deleted:
issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md is
what still stands once the general stop exists, under a slug that states it.

The boundary

kernel_exit_to_user_check is the one function every return to Ring 3 goes
through — the syscall gate, every device interrupt, the timer, the TLB IPI, the
trap epilogue and a task's first dispatch. A thread standing there holds no
kernel lock, has nothing in the block layer and nothing in flight on any
controller. Every record this kernel writes with a userland author is written
from inside a syscall, and a thread that never takes another userland
instruction never enters another syscall — so the stop is taken there, at the
same boundary a killed thread's last exit is taken at.

One rank, applied at two sites, with a test on each. A thread can stand at
that boundary carrying both marks. TaskShared::at_safe_point is the
declaration — written beside STOP and KILL themselves — and
scheduler::leave_ring3_if_due is one call and one match over it.
CpuSched::place applies the same rank to a task that is not running and reads
the two words itself, because the RT forward sits between the two arms. Both
are covered: stopping_outranks_killing_at_a_safe_point reds on the
declaration, stopping_outranks_killing on place's own order.

Tasks stop; CPUs do not

Every CPU keeps IF set, keeps taking its LAPIC timer and every device
interrupt and keeps taking passes — it has no userland left to dispatch. That is
what the USB stop below the last word needs and what klogd and iod need to
carry the log to its volume.

The carve-out is a capability, and the pid that was here is gone

wait_for_durable waits on one word, log::user::durable_ns, and that word
moves in exactly one place: a caller's clamped claim inside publish_durable,
reached only through SYS_LOG_READ. An earlier head took the carve-out's name
from there too — the pid that last advanced the word — and that was a defect.
A pid a caller nominates for itself through a syscall word is pid-as-authority,
self-claimed and refused by no name.
Any holder of Rights::LOG writes
cursor.durable, so a second logread holder could take the carve-out off
logd; stage one would then band logd and the wait that carve-out exists for
could not be satisfied.

The tension that put it there, stated rather than rediscovered. The first
implementation carved out the capability — every caller that passed
demand_syscap(_, Rights::LOG) — and it was refused twice, on two different
grounds. Mechanically: the set was a Lock<Vec<u32>>, unbounded, sized from
userland by a holder with dup, never pruned, behind a machine-wide lock taken
on every SYS_LOG_READ — a syscall logd polls for the life of the boot. By
width: ten committed configs grant logread to two programs

for f in $(git grep -l logread -- '*/system.toml' 'system.toml'); do
  grep -c 'syscap = \[.*logread' $f; done | grep -c '^2$'
10

and a holder joins the carve-out at its first SYS_LOG_READ. On
tests/testcases the second holder is test-runner, which reads inside its
log-gate and log-close builtins and writes files and spawns across the
sync_all the stop is taken before. On tests/metalcase test-runner holds
the right and its committed config gives it no job list — the image staged for
the T14 gives it the reboot builtin alone, which reads nothing — so it never
reads and stops in the first stage; the carve-out there is logd alone. The pid answered both objections at once —
one process, one word, no lock — and paid for it with the caller's word.

Both are answerable without paying that. The identity is the capability again,
so nothing a process says about itself joins it. The mechanical objections are
answered where they were made: LOG_HOLDERS is a fixed array of atomics — the
shape kernel/src/sched/kthread.rs's ROWS already uses to answer the same
kind of question about kernel threads — so there is no allocation, no lock, and
it is readable with preemption off inside the block layer; and a surplus holder
is refused by name and stops in the first stage like any other. A slot is never
given back: no pid is issued twice, so a dead holder's slot names nothing that
runs. An earlier head released slots at teardown, for a reason that was false of
this kernel — a dup records nothing, a holder is recorded at its first read, and
the tree's readers are logd, console, test-runner and quiesce_twice
and with no judge (the round-7 reviewer compiled the call out and the whole Fast
tier stayed green for it); the function and its call are deleted.
SYS_LOG_READ takes no lock of the carve-out's, as on main, and in the steady
state — logd finding itself in slot 0 — it is one relaxed load and no
read-modify-write.

An empty slot is u32::MAX, not zero. Zero is /system/bin/init's pid: the
process table issues from zero and init holds Rights::LOG, so with zero as the
sentinel holds_the_log(0) was false by construction and note_log_holder(0) a
silent no-op. u32::MAX is the pid percpu cannot represent, because it spells
idle with it; both functions refuse it by name, and quiesce's caller pair
starts there too.

The width is what is left, and it is tracked rather than claimed away. It is
narrower than main, where every process runs across the sync, and wider than
the wait, which depends on one. The exit condition is in
issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md: a
right of its own, moved in by init from a system.toml row naming the program
that writes /log, so the kernel still reads a capability and not a claim. That
is an ABI change and lands on its own pull request.

One check-then-act, tracked. block::counted() reads holder status when an
operation opens, and a process becomes a holder at its first SYS_LOG_READ from
any thread. A sibling's first read while this thread is inside a counted
operation at the instant the first stage's record is taken leaves in_flight = 1 on an N of N record — a false red, never a false green. A multi-threaded
holder is not hypothetical: test-runner on tests/testcases is one. What no
committed config arranges is its first read landing beside a sibling's
operation as the stop ends.
issues/kernel/the-stops-in-flight-count-decides-who-is-counted-at-the-open.md
carries it with its exit condition.

One shutdown, claimed by name, and judged

CALLER_PID/CALLER_TID are one global pair, and a second SYS_REBOOT or
SYS_SHUTDOWN caller overwrote the identity the first is exempt by. The first
caller can be parked — block::between_attempts parks inside the retry loop
writeback::drain_all runs — so a second caller's sweep bands the thread with
the rest of the shutdown to perform. quiesce::claim_the_shutdown is one
exchange (kernel/src/quiesce/claim.rs) taken before the watchdog is disarmed;
the loser is refused by name and gets AlreadyExists back through Ring 3.

That guard had no judge, and now has one. tests/quiescetwicecase runs
quiesce_twice. Its child closes a 64 KiB file on /log and asks for the
reset. writeback-stall parks iod, so that file's flush is the shutdown's own
drain's to find; the new quiesce-drain-refuse actuator refuses its FAT-1
mirror write eight times as a budget expiry on the thread running the shutdown,
so that thread parks in block::between_attempts for 10, 20, 40, 80, 160, 320
and 640 ms between its attempts. The job itself — a holder of the log
capability, so still running in the stop's first stage — reads the kernel's log
until the first refusal is in it and asks again from the other CPU, through
SYS_SHUTDOWN: the first seat is SYS_REBOOT, so the one boot judges the claim
on both power syscalls.
quiesce_refuses_a_second_shutdown judges where the kernel's refusal lands:

  [power] the second caller was refused at console line 225 while the first was in its ladder (refusals at lines 221 to 236):
    [kernel 0.442 cpu0] log-volume: quiesce-drain-refuse: refusing the shutdown drain's FAT-1 mirror write as a budget expiry, 1 of 8
    [kernel 0.442 cpu1] power: this machine is already stopping, so this caller stops with the rest

It asserts, in order: one Syncing filesystems... on the boot; eight actuator
refusals, the kernel's own count; the refusal by name once, between the first
and the last of them; the second caller's AlreadyExists reaching its console;
Rebooting.; and QEMU's guest-reset.

With the guard deleted it is red, in two shapes, and neither is "no
Rebooting.".
The prediction was that the banded first caller leaves the
machine with no last word. What the machine does instead is run a second
shutdown — on every one of the sixteen further boots that kernel was given at
40573079 (eight runs, each booted beside and then alone, EXIT=1 on all
eight; the round-7 reviewer added twenty, 38 of 38 with the handoff's). In ten
of the sixteen the second caller finds the first parked, bands it, and performs
the whole shutdown itself: two Syncing filesystems... on two CPUs — 1 ms apart
on eight of the ten, 2 ms and 11 ms on the other two — one stop: record and
one Rebooting., both the second caller's. In six it finds the first still
running between attempts and the two interleave:

[kernel 0.466 cpu0] stop: 3 of 3 userland thread(s) stopped across 2 cpu(s) in 0 ms over 1 sweep(s), 0 of 2120 userland block operation(s) still open
[kernel 0.466 cpu0] Rebooting.
[kernel 2.449 cpu1] Syncing filesystems...
[kernel 2.449 cpu1] stop: 3 of 4 userland thread(s) stopped across 2 cpu(s) in 2012 ms over 202 sweep(s), 0 of 2692 userland block operation(s) still open; this reset lands wherever the other 1 are
[kernel 2.449 cpu1] Rebooting.

— the second caller's stop spends its whole budget on the one thread that will
never reach a safe point, and then writes a sync line and a second last word
under the boot's last word. Both shapes write the sync line twice, which is
why that count is the judge's first assertion.

The stop leaves no filesystem update half made, and the volume checker says so

stop_if_blocked marks any parked thread, and block::between_attempts parks a
thread between two attempts of one filesystem update. A banded thread never
finishes its syscall, so what a refused attempt left on the volume stays there
at the reset. The round-7 reviewer named that as a hypothesis;
quiesce_leaves_the_volume_whole is the run.

quiesce-fsync-refuse, a third actuator of fat-mirror-write-refuse's shape,
refuses the active FAT's write in nine attempts of a SYS_FSYNC flush from
the moment the machine is stopping — after set_fat_entry has written the
mirror, which is the order that function keeps. The caller is logd. Its
ladder's parks (10 ms doubling, 2550 ms over nine refusals) outlast
log::SHUTDOWN_DURABLE, and a const assertion holds that; so the shutdown gives
up waiting for /log and the stop's second stage meets logd parked over
two FATs that disagree. The second stage and not the first, on purpose: /log
has 512-byte clusters, so in the first stage logd itself allocates the split
cluster within a few records and heals it by accident; below the second stage
nothing in the machine allocates again.

On the kernel without the fix — BUILD EXIT=0, TEST EXIT=1, the same
failure both times, from toyos-fat32-check:

FAIL quiesce_leaves_the_volume_whole: the machine's stop left the log volume breaking the format:
FAT 1 differs from FAT 0 at entry 44: 0x0FFFFFFF against 0x00000000. BPB_ExtFlags has mirroring on, so every copy must carry every update

The fix is the unit. The count the stop already keeps is per block
operation; what may not be cut is the filesystem's update. toyos_sched::task
carries MID_UPDATE in the task's state word, so stop_if_blocked refuses a
task parked inside an update in the same exchange that reads Blocked — a
flag beside the word would be a read and a mark that can come apart.
block::begin_update is the guard and ops::fsync holds it across its ladder,
the one retry ladder a userland thread can be parked in. The sweep counts such a
thread as running, so the stop waits for it inside PARK, bands it at the Ring 3
return of that same fsync, and says so in the record's shortfall clause if the
update outlasts the budget. With it — BUILD EXIT=0, TEST EXIT=0:

  [fat] the stop's second stage met a flush parked over split FATs and let it close; the checker is silent:
    [kernel 2.438 cpu0] shutdown: /log did not answer in 2000ms, so this shutdown's last lines are on the console only
    [kernel 3.006 cpu0] fsync: /log/2026-09-18-151803.log durable on attempt 10 after 2567ms — a refused attempt kept every page dirty and a later one delivered them

The judge asserts the harm first (the checker, over a partition asserted clean
before the boot), then that the arm fired nine times, that the shutdown gave up
inside that ladder, and that the flush closed after it — a whole volume that
does not say so was healed by something else, and is a red.

What it costs, said rather than hidden: that fsync: line is a kernel
record under the boot's last word, on a boot whose shutdown has already said its
last lines are on the console only.
issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md
carries it as what is still owed.

Read on the way, filed, and not fixed — it is main's and unmeasured:
toyos-fat32's append_cluster claims the new cluster and then links it, and a
refusal of the link leaves that cluster allocated where rollback_to — which
walks the chain from the file's head — never reaches it, so the caller's retry
allocates another. That is a leak a successful retry does not heal, and it is
the sentence log_flush_retry and fat_backing_revoked say in
issues/build/a-loaded-suite-reds-a-volume-checker-on-both-arms.md;
issues/filesystem/a-refused-link-write-leaks-the-cluster-append-cluster-just-claimed.md
names the host test that would show it. It is also why the actuator's first
refusal in an attempt is always the allocation's active-FAT write, which ends
the attempt: that is the one split set_fat_entry's order makes a retry heal.

The record, and the one counter behind it

stop: 9 of 9 userland thread(s) stopped across 2 cpu(s) in 30 ms over 4 sweep(s), 0 of 17401 userland block operation(s) still open

toyos-quiesce renders it and parses it and nothing else spells it: the kernel
writes it, src/metal.rs reads a Record off a stick and
tests/common/power.rs reads one off a console.
every_record_reads_back_as_itself_off_a_line_of_log is what a reworded
Display arm fails.

The stage split is gone. An earlier head kept two open-operation counters so
the second stage's record could carry the carve-out's own operations — and that
second term had no control that it ever counted. It is deleted rather than given
a judge, and Stage::covers, Stage::stopping, open_count and the guard's
stopped_by: Option<Stage> went with it. One counter answers the question: an
operation is counted iff the stop's first stage stops the thread that opened it,
decided at the open and carried on the guard as a bool. The second stage's
record prints the same word and the harness no longer asserts on it there: a
thread holding such an operation is inside a device with preemption off, which
the sweep counts as running, so that clause could never have fired before the
one above it.

The ceiling of 0 is a claim about a stopped machine, and it holds because no
open operation can outlive its opener's stop.
Lock::lock disables preemption
and the guard re-enables it on drop, and every begin_operation in the NVMe and
USB-storage paths sits inside a BlockDevice method reached only through
Partition::read_blocks/write_blocks/flush — under that guard, on the
opener's own thread, with no park anywhere in those wait paths. The one
begin_operation that is not inside a BlockDevice call is in
kernel/src/drivers/xhci/mod.rs's flush_disks: it runs on the reboot caller's
own thread and closes before either record is read, so it is outside both counts
by the time they are taken.

The two checks this change is judged by

Every mutation's build exit is given before its test exit — a mutation is a
measurement only once the mutated tree is shown to build. BUILD is
cargo test --test toyos-build -- --jobs 1 process_stats, which builds the
toolchain, the kernel, every guest binary and the image and then boots one
unrelated test; TEST is the same runner on the judge named. Measured on
this head's tree
, each restored from the index under a trap with the working
tree read back equal to it: the whole-change control, the whole of the
update-unit fix reverted, the holder call never taken, the guard deleted, and
the guard moved into one syscall. Measured at 40573079 by the implementer
and again by the round-7 reviewer, and not re-run here
: the two one-field
mutations of the block layer's count and the three-writer guest; nothing this
head changes is on their path.

The negative control: the whole change reverted onto the base, judges kept.
git restore --source=origin/main --worktree -- kernel/ toyos-sched/, with
origin/main at 9f91b581, puts the base's mechanism — 21 files — under this
branch's instruments; every judge, guest binary, boot config and profile row
stays the branch's (0 files differing under tests src toyos-quiesce issues).
kernel/src/quiesce.rs survives as a file the base's main.rs declares no
module for, so it is not compiled. BUILD EXIT=0, TEST
quiesce_stops_the_machine EXIT=1, red on the defect itself both times:

FAIL quiesce_stops_the_machine: 5 line(s) reached the console after the boot's last word:
  quiesce-writer: 5 2
  quiesce-writer: 4 2
  quiesce-writer: 0 5
  quiesce-writer: 3 4
  quiesce-writer: 2 5

The update-unit fix, reverted wholeMID_UPDATE out of the state word,
block::OpenUpdate gone, ops::fsync holding nothing, three files — BUILD
EXIT=0, TEST quiesce_leaves_the_volume_whole EXIT=1, the same failure
both times, quoted in the section above. Its independent oracle is
toyos-fat32-check, written from Microsoft's fatgen103 and sharing no code with
the driver it judges.

The independent oracle is the block layer's own count, which the stop does
not produce: it is incremented where an operation opens, in a driver leaf, and
the stop is judged against it. Both of its one-field mutations are seen.

Deleting the fetch_add alone, the guard still decrementing — BUILD EXIT=0,
TEST EXIT=1:

FAIL quiesce_stops_the_machine: the block layer still had 4294955142 operation(s) open on a thread this stop had stopped
  stop: 9 of 9 ... 4294955142 of 12154 userland block operation(s) still open

Making both ends agree that nothing is counted
(counted() && core::hint::black_box(false)) — BUILD EXIT=0, TEST
EXIT=1:

FAIL quiesce_stops_the_machine: this boot began no block-device operation on a stoppable thread, so the zero above is a counter that never counted
  stop: 9 of 9 ... 0 of 0 userland block operation(s) still open

The carve-out is load-bearing, and its removal is seen too — in the form that
builds.
Deleting the one call that records a holder does not build: the kernel
denies note_log_holder and REFUSED_A_HOLDER as dead code, so that mutation
measures nothing. The call compiled and never taken
(if core::hint::black_box(false) { note_log_holder(..) }) — the capability
still checked, no holder ever recorded — bands logd in the first stage, so the
durability wait can never be satisfied. BUILD EXIT=0, TEST EXIT=1, the
same failure both times:

FAIL quiesce_stops_the_machine: 1 line(s) reached the console after the boot's last word:
  [kernel 2.906 cpu1] shutdown: /log did not answer in 2000ms, so this shutdown's last lines are on the console only

The one-shutdown guard, whole and in both partial forms. The call, the
refusal, claim_the_shutdown, its static and its module deleted — BUILD
EXIT=0, TEST quiesce_refuses_a_second_shutdown EXIT=1, the same
failure both times:

FAIL quiesce_refuses_a_second_shutdown: this boot ran 2 shutdowns, not one: the second caller was let in
  [kernel 0.425 cpu0] Syncing filesystems...
  [kernel 2.438 cpu1] Syncing filesystems...

The claim moved out of quiesce() into sys_reboot alone used to pass,
because both callers used SYS_REBOOT. The second caller now asks through
SYS_SHUTDOWN, and that mutation is BUILD EXIT=0, TEST EXIT=1, the same
sentence both times. The judge counts the shutdowns before it asks how the
machine ended: the first run of that mutation read QEMU stopped this guest for "guest-shutdown" alone on one of its two boots, which is the second caller
winning — true, and not the harm.

A load and then a store in place of the one exchange is reached by no boot:
the second caller arrives a millisecond after the first. It is
kernel-loom/tests/shutdown_claim.rs's, over kernel/src/quiesce/claim.rs
itself and not a transliteration: EXIT=0 as written, EXIT=101 under
--features shutdown-claim-split with two callers ran the shutdown, wired
into host-tests.yml beside the other model controls.

The workload judge fires on the partial fix that used to pass. It asked for
at least six threads and was met by three writers plus three others. It now asks
for exactly nine: six writers, init, and test-runner's main and deadline
threads. logd holds the log and the first stage's sweep passes over it
uncounted; the job's own main thread is the caller. Measured at 40573079, by
the implementer and again by the round-7 reviewer — the guest run with three
writers where the harness is told six, BUILD EXIT=0, TEST EXIT=1, stop named 6 ... make 9, the same failure both times. This head changes that
message's words and not its count.

The green arm was re-measured after the last restore: BUILD EXIT=0, all
three judges EXIT=0, with the record, the refusal and the closed flush
quoted above.

The second oracle is the machine: a T14 readback with Rebooting. last and the
stick enumerating 0 s after, which is owed and named below.

The guest says when it is ready; the harness counts what the kernel stopped

quiesce_writers puts six threads into a write-and-fsync loop and asks for the
reset only once every one of them has completed a pass. The harness judges the
kernel's own thread count, which the guest cannot write, and judges it exactly:
a writer whose loop ends on an I/O error now says which call failed, and leaves
eight where nine are asked for.

Gates, each with the exit status of the command itself

All on this head's tree, 2026-09-21.

command result
cargo test --test toyos-build (the harness's whole Fast tier) 346 passed, 1 failed, 347 total, 102 held back for the nightly, 412.3 s, EXIT=1
cargo test --lib 307 passed, 1 failed, 1 ignored, EXIT=101
cargo test --workspace --exclude toyos-build 142 test result: ok lines, 0 failed, EXIT=0
toyos-sched alone 74 unit tests passed, the new a_task_parked_mid_update_refuses_the_parked_mark among them, EXIT=0
kernel-loom shutdown_claim / the same under --features shutdown-claim-split EXIT=0 / EXIT=101
the host job's five cargo clippy invocations, spelled as src/clippy.rs spells them EXIT=0 on each
the three quiesce judges after the last mutation's restore EXIT=0 on each

cargo test --lib's red is main's, and it is a date.
redlist::tests::every_row_can_say_what_it_claims refuses four rows measured on
2026-08-20 and 2026-08-21 as more than 31 days ago, and still standing
(console_line_atomicity, kill_while_blocked twice, xhci_full_speed_device).
This branch does not touch src/redlist.rs (git diff origin/main -- src/redlist.rs is empty), so origin/main at 9f91b581 reds the same way
today, and a plain cargo test stops at that target before it boots a guest —
which is why the tier above is run by its own target. The rows are their
owners' to re-take, retire or delete; nothing here ages them away.

The tier's one red is not shown to be this diff's, and it is not on the
list.
handle_kill_policy: 16 more killed processes left more live objects behind: [("SharedMem", 5, 6)], then ALONE handle_kill_policy: GREEN, in a
tier that took 412.3 s against the 147.3 s the same tier took at 40573079
another agent's work resumed on this host beside it, and the same boot shows
usb-storage: ... no answer in the data phase in 2000 ms. cargo run -- --known-red handle_kill_policy answers NOT KNOWN-RED: the redlist carries
this sentence (with Process where this one says SharedMem) as a loaded-host
sighting retired on 2026-09-04 by six green tiers. It is a machine-wide census
either side of a kill on a shared boot; nothing this branch changes allocates or
frees a SharedMem. One sighting prices nothing, and it is the redlist's to
re-open at its owner, not re-run away here. fs_dirs_durable, the tier's red at
40573079, was green in this run.

Two reds seen at earlier heads, named rather than re-run away.

  • blocked_dump — the round-6 reviewer's tier run at c2352819 read PANIC: ... src/sched/dump.rs:134:5: the blocked-task dump ran under a lock: preempt depth 2, backtrace dump::request <- driver::drain_irqs <- driver::pass_block <- completion::wait_inner <- inbox::submit, green alone. It did not occur in
    this head's tier run. The path is main's: this branch's only change under
    kernel/src/sched/ is 14 lines in driver.rs, and completion.rs,
    dump.rs and inbox.rs are untouched. It is a kernel defect on a path this
    branch does not own and belongs to its owner in the redlist.
  • guest (11), CI run 34896137159 at 8fdb28b9: FAIL metal_device_probe: usbwrite: the job refused — IoFailed, green alone. The diff does touch the
    guard every USB write opens (block::begin_operation) and the placement every
    wake crosses (CpuSched::place), so it was measured rather than read: a
    same-session A/B on metal_device_probe, three boots at 40573079 and three
    with the working tree set to origin/main at dc38a054 (0 files differing
    from it) — EXIT=0 on all three at head, EXIT=0 on all three on main.
    CI's own run at c2352819 (35100889732) concluded success with that shard
    in it. The round-6 reviewer found the line in one of the last 40 failed
    ci.yml runs and in no other; here it reproduced on neither arm. Six boots do
    not price a rate that small, so this says the diff is not shown to cause it,
    not that it cannot; it is the redlist's to carry at its owner.

The merge, and the numbers it had to choose between

origin/main at dc38a054 is merged in, 7 commits past the previous merge
base, with no conflict. The earlier merge of 45 commits at 8261079e had three
conflicted files, every hunk of both sides accounted for in that merge commit's
message, and one choice that still stands: boot.deadlinewedge.complete_ms,
.back_secs, .deadline_lateness_ms, boot.hardlockup.complete_ms and
.back_secs carry a measured both sides moved — main's from T14 run 44, this
branch's from run 46. measured is the last reading the machine gave, so run
46's stand; main's run-44 panel_max_us/panel_us on those same two boots are
untouched by this branch and stay. The run-56 staging below supersedes all of
them at once.

The machine

The four T14 boots are owed at this head, and the arms are the
orchestrator's. The run-55 images were staged at 40573079 and are stale: this
head changes the kernel every one of them carries (block.rs, object/ops.rs,
quiesce.rs and its new claim.rs, log/user.rs, process.rs,
fat32_adapter.rs) and the scheduler's state word. A run-55 reading still
describes this head in one respect only — the first stage's record on a boot
where nothing is parked in an fsync ladder takes the same path — and it does
not describe the second stage, which now waits such a ladder out. They are
restaged at target/metal-quiesce-run56/ from a tree clean before and after at
this head, with hashes, invocations and expected lines in
t14-request-quiesce-6.txt. Each of the four carries the stop record's literal
once; an image from a worktree without this branch carries it zero times and
Rebooting. once, which is what says the reader works.

What this costs, declared

tests/metal-profile.toml gains 15 rows — park_open_operations for each of
the 15 boots that reset through quiesce. park_ms is priced nowhere and that
is the point: a stop that gives up spends its budget and no more, so a ceiling
on the duration could be exceeded only in the expiry case, which
Readback::stop_completed already reds on for every boot the suite reads back
by reading the record's own shortfall clause.

Three registered names. quiesce_stops_the_machine is priced: CI run
34896137159 measured it at 4,434 ms across twelve shards, inside
FAST_COMMIT_MS, so it stays Tier::Fast. quiesce_refuses_a_second_shutdown
is priced by CI run 35355556050 at 40573079, whose test-durations-merged
artifact reads 4,249 ms across twelve shards; that one row is taken from it
and no other, and it stays Tier::Fast. quiesce_leaves_the_volume_whole is
new and carries the UNMEASURED marker, so this landing pays the two CI cycles
a new name costs a second time; the dev host ran it in 4 s, most of it the
2,000 ms the shutdown waits for a /log the actuator keeps from answering.
quiesce_writers and quiesce_twice are guest binaries on RUST_SKIP and
carry no row. Two new actuators, quiesce-drain-refuse and
quiesce-fsync-refuse, each with a FLASHABLE ruling of Never, and one
kernel-loom control, shutdown-claim-split. quiesce::PARK says what it is —
a budget, not a bound the kernel can prove — and its expiry is a clause in the
record rather than a hang.

Decisions in toyos-quiesce/ (pure, 10 tests); effects in
kernel/src/quiesce.rs.

Filed and not fixed:
issues/build/two-worktrees-race-provisioning-the-toolchains-cargo.md,
issues/kernel/the-machine-stops-without-saying-which-thread-took-the-time.md,
issues/diagnostics/a-t14-wedge-ran-the-deadline-out-and-sealed-nothing.md,
issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md,
issues/kernel/the-stops-in-flight-count-decides-who-is-counted-at-the-open.md,
and — in main's own code —
issues/kernel/the-shutdowns-drain-counts-the-queue-while-iod-holds-an-entry.md
(writeback::drain_retrying counts the queue without the VFS lock drain_one
pops under, so a shutdown can sync and reset while iod still owes a
budget-refused flush) and
issues/filesystem/a-refused-link-write-leaks-the-cluster-append-cluster-just-claimed.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK

Japabu and others added 5 commits September 13, 2026 16:47
…thread

`issues/kernel/quiesce-runs-while-userland-still-does-io.md` recorded two
one-off T14 boots of `tests/metaldevicecase`. Every T14 boot of
`tests/metalcase` reproduces it: that config starts `sshd`, `sshd` spends a
second retrying a `netd` that has no NIC on this machine, and it is still
spinning in Ring 3 when the `reboot` job asks for the reset — so its own
`exit:` record lands after the boot's last word.

    grep -n "Rebooting\.\|exit: sshd" <readback>/kernel.log   (EXIT=0)

    readback   Rebooting.   exit: sshd   gap     sshd cpu
    metalcase  6.903 cpu4   7.033 cpu1   130 ms  2380 ms
    run 34     6.101 cpu4   6.366 cpu1   265 ms  2382 ms
    run 35     6.929 cpu4   7.035 cpu1   106 ms  2325 ms

On run 35 `sshd` burned 2,325 ms of CPU across the 2,326 ms between its spawn
at 4.709 and its exit, so it was never blocked: one CPU kept dispatching
userland across `Rebooting.`. All three came back with `stick_secs 0`, so the
device half stays answered and only the ordering half is open.

Which also names the asymmetry the fix has to live with: `bootlog::verdict`
reds on all three and `tests/metal-profile.toml` prices no number that would
notice.

`cargo test --lib`: 295 passed, 0 failed, EXIT=0.
`cargo test --workspace --exclude toyos-build`: 138 `test result: ok` lines, no
failures, EXIT=0.

The kernel half of this task is not here: the shared sysroot is claimed by
`toyos-aperture` for `host-bridge-abi`, so no kernel builds in this worktree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`quiesce` disarmed the watchdog, synced every filesystem and wrote the boot's
last word while every other CPU was still running userland. Both of those are
claims about a machine, and they were true only of the instant that made them:
on every T14 boot of `tests/metalcase`, `sshd` — spinning in Ring 3 retrying a
`netd` that has no NIC on that machine — wrote its own `exit:` record 106 to
265 ms after `Rebooting.`, and `bootlog::verdict` was EXIT=1 for it.

So the machine is stopped first, and the stop is taken at one place.

**The boundary.** `kernel_exit_to_user_check` is the one function every return
to Ring 3 goes through — the syscall gate, every device interrupt, the timer,
the TLB IPI, the trap epilogue and a task's first dispatch. A thread standing
there holds no kernel lock, has nothing in the block layer and nothing in
flight on any controller. Every record this kernel writes with a userland
author is written from inside a syscall, and a thread that never takes another
userland instruction never enters another syscall — so `quiesce::stop_here_if_due`
goes there, beside `exit_if_killed`, which stands there for the same reason.

**The band.** `toyos-sched` grows a third container beside `dying`: a `stopped`
band no pick serves and nothing is ever taken out of. A running thread enters
it through its own `dispose_stop` at that boundary; a thread *parked* inside a
syscall is marked where it lies by `stop_if_blocked`, whose CAS refuses a
running task precisely because a running one may hold a lock. `place` — the one
funnel every wake, adopt and retire goes through — routes anything carrying the
mark into the band ahead of the RT forward and ahead of the kill check, so a
deadline firing after the stop cannot put a thread back on a run queue to
finish its syscall and log. That last arm is what makes `Rebooting.` last by
construction rather than by luck, and it is sound because a thread that parked
holds no lock: `kernel/CLAUDE.md` already states that invariant.

**Tasks stop; CPUs do not.** Every CPU keeps `IF` set, keeps taking its LAPIC
timer and every device interrupt and keeps taking passes — it has no userland
left to dispatch. That is what the USB stop below the last word needs and what
`klogd` and `iod` need to carry the log to its volume. `deadline::this_cpu`'s
freeze-inside-a-pass is the wrong shape here: it would strand whatever lock the
thread on that CPU held, and `sync_all` is the first thing that would wait.

**The carve-out** is `logd`, and the kernel had no name for it. `wait_for_durable`
waits on a word any `Rights::LOG` holder publishes, so `log::user` now records
the pid of whoever *moved* that word and `Stage::ExceptLog` leaves that one
process runnable until the wait returns. Then `Stage::All`, then the reset. A
machine where nobody ever claimed durability carves nothing out and ends on
`wait_for_durable`'s existing two-second budget, with the record it already
writes.

**The record**, beside `irq:` and `nvme:` and above the last word:

    stop: 9 of 9 userland thread(s) stopped across 2 cpu(s) in 0 ms over 14
    sweep(s), 0 block operation(s) still open

The open-operation count is the block layer's own, taken inside `block::begin_operation`
rather than derived from the stop, so the two can disagree. The bound is
`quiesce::PARK` — one `QUANTUM_NS` plus one `block::OPERATION`, the two things a
thread that must stop can be doing — and its expiry is a clause in the same
record, never a hang.

Decisions in `toyos-quiesce/`: which thread must stop, when the stop is over,
and what the record says. Effects in `kernel/src/quiesce.rs`.

THE TWO CHECKS, both run:

* Negative control — the whole change reverted onto the base, with the test
  kept: `quiesce_stops_the_machine` reds on lines reaching the console after
  the boot's last word. Unit-level, `place`'s stop arm deleted and nothing
  else: `a_wake_for_a_stopped_task_lands_in_the_band_and_not_the_run_queue`
  reds `the wake reached the band: left 0, right 1` (EXIT=101).
* Independent oracle — the block layer's count, which the stop does not
  produce; and the machine, whose readback is requested separately.

GATES

    cargo test --lib                                  295 passed, EXIT=0
    cargo test --workspace --exclude toyos-build      EXIT=0
    cargo test quiesce_stops_the_machine              1 passed, EXIT=0
    cargo test in toyos-sched/                        72 passed, EXIT=0

`quiesce_stops_the_machine` and `quiesce_writers` are new registered names and
carry the `UNMEASURED` marker, which is provisional by declaration: the price
gate reds on this commit, CI measures them, and the next commit replaces the
markers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`quiesce_stops_the_machine` asserted that nothing follows `Rebooting.` and that
assertion was green on the base as well: the writer threads produce no kernel
record, and under QEMU nothing else was left to write one. A judge that cannot
fail is not a judge.

So each writer prints one line per pass. Six of them, with `quiesce-late-word`'s
staged hundred milliseconds of shutdown still to run, put lines under the boot's
last word on a machine that was not stopped — and on one that was stopped not
one of them can, because a stopped thread takes no further userland
instruction.

    cargo test quiesce_stops_the_machine   1 passed, EXIT=0
    [power] the machine stopped before it claimed anything:
    [kernel 0.909 cpu0] stop: 9 of 9 userland thread(s) stopped across 2 cpu(s)
    in 0 ms over 20 sweep(s), 0 block operation(s) still open

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Seen twice in one session on a host running ten worktrees: `cargo test <name>`
panicked at `src/toolchain.rs:940` on a toolchain directory another worktree's
exclusive `give the toyos toolchain its own cargo` phase was midway through
writing. The refusal is right about the state and wrong about the cause, and a
plain re-run passed both times with nothing changed.

Filed, not fixed: it is the build system and not this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
@Japabu Japabu changed the title The quiesce defect has a standing reproduction, and it is a spinning thread The machine stops before Rebooting. claims anything about it Sep 13, 2026
Run 41 booted all four arms on 52c8a3f and the driver's verdict was EXIT=0 on
every one, with the boot stick enumerated 0 s after each.

`tests/metalcase` is the boot this branch exists for. Every earlier readback of
it put `exit: sshd pid=8 code=0` 106 to 265 ms after the boot's last word; run
41's kernel log carries **one** `exit:` record in the whole boot — `netd`, at
4.405 s — and ends:

    [7.955 cpu4] stop: 7 of 7 userland thread(s) stopped across 8 cpu(s) in
    129 ms over 67953 sweep(s), 0 block operation(s) still open
    [7.955 cpu4] usb-quiesce: disk 0 implements no SYNCHRONIZE CACHE, so it owed none
    [7.955 cpu4] Rebooting.

`sshd` never exited at all: the stop caught it alive and it took no further
userland instruction.

`tests/metaldevicecase` — megabytes written and read back before the reset —
stopped 5 of 5 in 0 ms over one sweep, `Rebooting.` last, stick at 0 s on the
tightest ceiling in the file.

`deadlinewedge` and `hardlockup` are unchanged and carry **no** `stop:` record,
which is what their reset path predicts: both go through `acpi::reset_now` and
neither calls `quiesce`. A record on either would have been a finding.

Measured, into the rows that had no reading and only those:

    boot.metalcase.park_ms                        129   (ceiling 2010)
    boot.metalcase.park_open_operations             0   (ceiling 0)
    boot.metaldevicecase.park_ms                    0
    boot.metaldevicecase.park_open_operations       0
    boot.metaldevicecase.complete_ms             1255
    boot.metaldevicecase.back_secs                 51
    boot.metaldevicecase.stick_secs                 0   (ceiling 5)

The ceilings do not move. `quiesce::PARK`'s derivation is the bound; 129 ms is
one workload's one reading and the file's own rule is that `measured` is never
the gate.

`issues/kernel/quiesce-runs-while-userland-still-does-io.md` is deleted: the
device half was already answered and the ordering half is now answered on the
machine. Its one citation, in
`issues/hardware/a-t14-boot-wedges-after-a-jobs-exit-and-nothing-said-why.md`,
is rewritten rather than dropped — that issue's `hand_back` candidate is
narrowed by the stop and not eliminated by it, and it now says so without
pointing at a file that is gone.

What run 41 found that this branch does not fix, filed at
`issues/kernel/the-stop-sweeps-at-memory-speed-and-names-no-thread.md`: 67,953
sweeps in 129 ms is 1.9 us apiece, each taking the machine-wide process table
lock, so the loop may be contending with the thread it is waiting for — and the
record names a count and never a thread, so the log cannot say which of the
seven took the time.

    cargo test --lib   295 passed, 0 failed, EXIT=0

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
@Japabu
Japabu marked this pull request as ready for review September 14, 2026 07:03
Japabu and others added 16 commits September 14, 2026 09:56
…and its sweep has a cadence

The review of PR #452 sent the branch back. This answers it.

The carve-out was minted from `cursor.durable`, a value any `Rights::LOG`
holder writes, so whichever program last moved the word decided which process
survived a stop-the-world; five committed configs grant `logread` to two
programs, and in `logrotatecase` the second holder also holds `power`. The
identity now comes from where the capability was demanded — `sys_log_read`
records its caller after `demand_syscap` succeeds — and the carve-out is every
holder rather than the newest claimant, so an impostor can join it and never
take it off `logd`. `Stage::ExceptLog` carries no pid; `Thread` carries the
fact, and `DURABLE_PID` is gone.

The sweep polled at memory speed: 67,953 sweeps in 129 ms on the T14, taking
the machine-wide process table each time and contending with the very threads
it waited for. It now waits one `QUANTUM_NS` between sweeps. The guest reads
2 sweeps where it read thousands, and the half of the tracked weakness that
survives — the record names a count and never a thread — is narrowed to that.

`Record`'s wire form is rendered and parsed in `toyos-quiesce` and nowhere
else: `src/bootlog.rs`'s second spelling of `stop: ` is deleted, `src/metal.rs`
reads a `Record` rather than splitting strings, and the harness's three
constants are gone. A round-trip test fails on a reworded `Display` arm.

The second stage discarded a `#[must_use] Record` and nothing tested it at all.
It runs below the boot's last word, so its record goes on the black box, which
is the one channel a reset's own account has; `power::done_chain` reads it back
off the page after the seal, and deleting the call reds `blackbox_done_chain`
on both the guest and the bench.

`OPEN_OPERATIONS` counted operations `iod`, `usbd` and `xhci::flush_disks`
begin, which the stop never stops, so its ceiling of 0 judged something the
stop does not control; it now counts only the threads the stop stops. It was
also asserted in its zero state alone: the record now carries how many such
operations the boot began, and the guest test refuses a zero that is a counter
which never counted (14,859 on that boot).

The rest, each from the review: the metal suite could not see a stop that gave
up, because an expiry spends its budget and reads under its own ceiling —
`Readback::stop_completed` reads the shortfall clause for every boot; the
headline judge passed vacuously on a boot that never wrote `Rebooting.`; the
caller's `map_or(0, 0)` silently exempted whichever thread held those ids and
is now a refusal by name; the kick-every-other-CPU loop was a third copy and is
`apic::kick_all_but_self`, with `deadline.rs` and the panic drain on it;
`stopped_len()` had no caller and is now the `sched:` census's third band; the
actuator's doc named a test that does not exist. `toyos-sched/sim` did not
compile against `Container::Stopped`, which no gate on this branch had run.

`boot.metalcase.complete_ms` and `back_secs` were stale against run 41 and are
its readings, 1260 and 45. The four `measured` fields on the park rows are
dropped: they were read off a kernel whose record wording and sweep cadence
this commit changes, and no machine has answered the new one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Four arms on the T14 at 22258c9, EXIT=0 on every one, the stick enumerated
0 s after each. `tests/metal-profile.toml` takes their readings.

    metalcase        stop: 7 of 7 userland thread(s) stopped across 8 cpu(s)
                     in 70 ms over 8 sweep(s), 0 of 9137 userland block
                     operation(s) still open        (kernel.log:330)
    metaldevicecase  5 of 5 ... in 0 ms over 1 sweep(s), 0 of 14192 ...
                                                    (kernel.log:406)

`Rebooting.` is the last line of both, two lines under the record. Neither
wedge boot wrote one, which is what they owe: both reset through
`acpi::reset_now` and `quiesce` is not on that path.

Against run 41's pre-cadence kernel on the same config: 129 ms over 67,953
sweeps becomes 70 ms over 8. The sweep is no longer contending with the
threads it waits for, and what is left is one thread's remaining syscall —
which is the reading
`issues/kernel/the-machine-stops-without-saying-which-thread-took-the-time.md`
says the record cannot attribute to a thread, and still cannot.

Every `measured` field these four boots produce is run 46's, not some of
them: `measured` is the last reading the machine gave, and updating one boot
and not its neighbours is the finding this branch was already sent back for.
metalcase 1260 -> 1199 ms and 45 -> 51 s, metaldevicecase 51 -> 65 s,
deadlinewedge 1199 -> 1254 ms, 236 -> 231 s and 61 -> 60 ms late,
hardlockup 1201 -> 1206 ms and 172 -> 170 s. The four park rows are priced
for the first time.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
… ran

The safe point checked the kill mark before the stop mark, against what
toyos-sched declares and what `stopping_outranks_killing` asserts: a thread
carrying both took `Dispose::Exit` and unwound into `process.rs`'s
`log!("exit: ...")` — the record this branch exists to keep out from under the
boot's last word.

`TaskShared::at_safe_point` is now that rank, declared beside `STOP` and `KILL`
in the scheduler core and read by `scheduler::leave_ring3_if_due`, which is one
call and one match at `kernel_exit_to_user_check`: `exit_if_killed` and
`stop_current` are gone, so the two sites that could disagree are one site with
no order in it. `stopping_outranks_killing_at_a_safe_point` reds on the
declaration. `object::ops`'s fsync retry loop asked the same question about the
kill mark alone and now asks this one, so a stopping thread stops retrying at
once instead of spending a block-layer deadman inside the park.

`quiesce_stops_the_machine` asserted nothing about its six writers: a boot whose
workload died left nothing under the last word either and passed every judge.
It now refuses a boot in which any writer wrote nothing *above* the last word,
by that writer's own line, before it reads what follows it.

`LOG_READERS` was appended to and never removed from; `teardown_bookkeeping`
now forgets an exiting pid, so the carve-out set is bounded by the live process
count and an exited pid holds no carve-out.

The sweep counted a thread between its table insert and its task mint in
neither band; it is counted running, so the record cannot say "N of N stopped"
with a thread still to be dispatched.

`Progress`'s three variants had one consumer reading one bit of them:
`Sweep::keep_waiting` replaces it. `Stage::must_stop` takes a `ThreadId` where
it took an untyped pair, and `Thread` carries one.

Prose deleted, not rewritten: block.rs's false claim about `iod`/`usbd` (the
xHCI flush runs on the caller's own userland thread and is counted),
`toyos-quiesce`'s "every process that can satisfy the wait" (the set is every
process that has *read*), `apic.rs`'s title line, quiesce.rs's copy of it,
`stop_current`'s fourth spelling of the boundary invariant, machine.rs's
appended narration, power.rs's four-bullet restatement of the four checks
beneath it and its copy of machine.rs's page-record contract,
quiesce_writers.rs's copy of that, tests/toyos.rs's transient UNMEASURED note,
and the sentence about "the two lateness fields" that this branch's two park
fields made false.

`idle_is_spinning`'s fixtures spell the `stopped=` field the kernel writes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`writer(s) [1, 2, 3, 4, 5] of this boot's 6` rather than five copies of the
line prefix with the match's trailing space in each.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`object::ops`'s fsync retry loop asked the kill mark, and round two widened it
to both marks. That widening is withdrawn: a thread inside a sync may be
abandoned when it is killed, because its caller is gone, but not when the
machine's stop names it — `quiesce` stops userland and then claims every
filesystem is synced, and a sync the stop cut short is a claim about work that
did not finish. Stage two runs below `sync_all`, so a `logd` fsync abandoned
there had nothing left to flush it.

The rank at the Ring 3 boundary is untouched: `leave_ring3_if_due` is still one
call and one match over `TaskShared::at_safe_point`, which is what the review
asked for. `driver::current_kill_pending` comes back for the one caller that
wants the kill mark alone.

Run 48 of the T14 is what prompted this, and it did not prompt it for the
reason it looked like. The full account is
`issues/diagnostics/a-t14-wedge-ran-the-deadline-out-and-sealed-nothing.md`:
`tests/metalcase` wedged after `exit: netd` at 3.7 s, ran its 120 s boot
deadline out, sealed no record, and was reset without a sync — which is what
tore the volume the checker refused. The partition carries no `Syncing
filesystems...`, no `stop:` and no `Rebooting.`, so the shutdown's stop never
ran on that boot and the torn FAT is downstream of the wedge. The shape is the
one `issues/hardware/a-t14-boot-wedges-after-a-jobs-exit-and-nothing-said-why.md`
records, whose exit condition this occurrence still did not meet.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
…y right-holder

The first stage exempted every process that had ever called `SYS_LOG_READ`
holding `Rights::LOG`. Ten committed configs grant `logread` to two programs,
nine of them under `tests/` (`for f in $(git grep -l logread -- '*/system.toml'
'system.toml'); do grep -c 'syscap = \[.*logread' $f; done | grep -c '^2$'`), so
on `tests/testcases` and `tests/metalcase` the carve-out was `logd` *and*
`test-runner` — which writes files and spawns. The test runner therefore kept
running across `sync_all`, which is the hazard stated in the comment directly
above that line, and nothing synced after `Stage::All`.

The wait is what names the carve-out. `log::wait_for_durable` polls
`log::user::durable_ns`, and that word moves only where a caller's claim
advances it, so the pid that advanced it is recorded beside it and the carve-out
is that one process. A reader that has published nothing has ended nobody's wait
and stops in the first stage with everything else; a machine where nobody has
published carves nothing out and ends on the wait's existing two-second budget
with the record it already writes. On every committed config the publisher is
`/system/bin/logd` and nothing else.

Two atomics and one relaxed store where the word actually moves, so the
machine-wide `Lock<Vec<u32>>` is off `SYS_LOG_READ` — a syscall `logd` polls for
the life of the boot — and the set that had to be pruned at process teardown
goes with it.

What the one carved-out process can still do between `sync_all` and the reset is
`issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md`: it
writes to an open file that no sync of the kernel's reaches, and a syscall it is
already inside when the second stage begins runs to its end. That is what is
left of the deleted `quiesce-runs-while-userland-still-does-io.md` once the
general stop exists, and it is narrower than that file's claim in the one way
that matters: no process the wait does not depend on runs at all.

`quiesce::PARK` now says what it is. One syscall may open several block-layer
operations in a row and `block::DEADMAN` is what bounds that sequence, so the
budget was never the bound its derivation claimed. The fifteen `park_ms` rows go
with the claim: a stop that gives up spends its budget and no more, so that
ceiling could only be exceeded in the case `Readback::stop_completed` already
reds on for every boot. `park_open_operations` stays — the block layer produces
it, so it is the one number that can contradict the stop.

Deleted prose: the citation refuted by `tests/metalcase/system.toml:57`, two
tracker citations that resolve to nothing from a clean checkout, the argument
for another module at `block.rs`'s statics, the restatement of `_deadline`, and
the sentence a round-2 deletion left broken in `tests/common/metal.rs`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
… they are ready

Four of six runs of `quiesce_stops_the_machine` at this head red in three modes,
recorded in the branch's staging record for run 52. Two of the three were judges
asserting what the design does not promise. None was the stop.

**The open-operation count included the one process the stop deliberately leaves
running.** `block::begin_operation` counted every operation opened by a userland
thread, and the record reads that count at the end of stage one — where the
carve-out, the process the durability wait depends on, is still running by
construction. The `1 of 8466` recorded at this head is that process inside a
device transfer: the machine working exactly as the stop intends, judged against
a ceiling of 0.

An operation exists only while its opener is executing inside the device. Every
`begin_operation` is a driver leaf (`nvme`, `usb_storage`, `xhci`), its guard
lives inside one `BlockDevice` call, and no disk wait in this kernel can park —
so a thread the stop has stopped holds none, and that, rather than "userland
holds none", is what the block layer can contradict the stop with. The count is
now kept per stage: an operation is closed by the stage that stops the thread
that opened it, decided at the open and carried on the guard so the two ends
cannot disagree. `Stage::covers` is the one declaration and `must_stop` and
`Stage::stopping` both read it. The ceiling of 0 does not move, because it is
now the claim the design actually makes; stage two counts the carve-out's
operations as well, and `done_chain` reds on a non-zero there, which is the only
judge that sees the log writer's own operations closed.

**The workload's readiness was a sleep.** `quiesce_writers` spawned six writers,
slept 300 ms and asked for the reset, and the judge then required every writer's
own line above the boot's last word. Under load a writer had not been scheduled
inside that window, so the judge red on a precondition the guest never
established. The writers now say so: each signals after its first completed
write-and-fsync pass, and the reset is asked for once all six have. A writer
that never reaches its loop is now one line naming how many did.

The park's budget does not move. Twelve boots at this head measured it at 10-169
ms against `quiesce::PARK`'s 2,010 ms, on a quiet host and under 24 spinners
alike, and the worst of them is the quiet host's. The readings and the reason a
wider budget is not free — `block::DEADMAN` is 120 s and the boot deadline runs
straight through `quiesce` — are in
`issues/kernel/the-machine-stops-without-saying-which-thread-took-the-time.md`.

Measured here, all at feffffb before this commit: five quiet-host runs green;
four loaded-host runs, three red, every red the writers' precondition and every
one of them carrying `9 of 9` stopped and `0` open.

Gates, each with the exit status of the command itself:

    cargo test -p toyos-sched -p toyos-quiesce      73 + 11 passed          EXIT=0
    cargo run -- --build-only                                              EXIT=0
    cargo test --lib                               295 passed, 1 ignored   EXIT=0
    cargo test --workspace --exclude toyos-build   140 ok lines, 0 failed  EXIT=0
    cargo run -- --clippy                                                  EXIT=0
    cargo test --test toyos-build -- quiesce_stops_the_machine x3          EXIT=0

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
…itness goes

The fifth review of PR #452 sent the branch back on a pid. This answers it.

`DURABLE_PID` was minted from `cursor.durable`, and that pid decided which
process survived a stop-the-world and ran across `Rebooting.`. Any holder of
`Rights::LOG` writes that word, so a second `logread` holder calling
`publish_durable` took the carve-out off `logd`; stage one then banded `logd`
and the wait it exists for could not be satisfied. Ten committed configs grant
`logread` to two programs. A pid a caller nominates for itself through a
syscall is pid-as-authority, self-claimed and refused by no name, in a tree
where a process holds exactly what its parent moved into it.

**Why the pid came back, stated as the tension it was solving.** Round 1 struck
it and `22258c9c` put the carve-out on the capability: every caller that passed
`demand_syscap(_, Rights::LOG)`, kept in a `Lock<Vec<u32>>`. Round 2 refused
that set — unbounded, sized from userland by a holder with `dup`, never pruned,
and a machine-wide lock taken on every `SYS_LOG_READ`, a syscall `logd` polls
for the life of the boot. Round 3 refused its width — on `tests/testcases` and
`tests/metalcase` the second holder is `test-runner`, which writes files and
spawns across the `sync_all` the stop is taken before. The pid answered both at
once: one process, one word, no lock. It bought them with the caller's word,
which is the one price this tree does not pay.

Both are answerable without it. The identity is the capability again, so
nothing a process says about itself joins it, and the two mechanical defects are
answered where they were: `LOG_HOLDERS` is a fixed set of atomics, the shape
`sched/kthread.rs`'s `ROWS` already uses for the same question about kernel
threads — no allocation, no lock, readable with preemption off inside the block
layer — and a surplus holder is refused by name and stops in the first stage
like any other. `forget_log_holder` releases a slot at teardown, because a pid
is issued again. The width is what is left: it is narrower than `main`, where
every process runs across the sync, and wider than the wait, and it is recorded
at `issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md`
with its exit condition — a right of its own that `init` moves in, which is an
ABI change and lands on its own pull request.

The stage-two block count had no control that it ever counted. Deleting
`+ OPEN_LOG_OPERATIONS.load(...)` left every judge green, because the second
record is asserted only in its zero state and `begun` summed both buckets. It
is deleted rather than given a judge, and with it `Stage::covers`,
`Stage::stopping`, `open_count` and the guard's `stopped_by: Option<Stage>` —
an abstraction feeding one uncontrolled assertion. One counter answers the same
question: an operation is counted iff the stop's first stage stops the thread
that opened it, decided at the open and carried on the guard as a bool. Every
one-field mutation of what is left is now seen: the guard still decrements, so
deleting the `fetch_add` underflows the count and the guest judge reds on a
non-zero `in_flight`; deleting the counting altogether reds on `begun == 0`.

A second `SYS_REBOOT`/`SYS_SHUTDOWN` caller overwrote the identity the first is
exempt by, and the first can be parked in `block::between_attempts` inside the
`sync_all` it is waiting on — so the second sweep banded the thread with the
rest of the shutdown to perform, and that reset was never reached. The machine
has one shutdown, claimed by name; the loser is refused, returns, and is banded
at its own safe point like every other thread.

The guest guaranteed six writer lines above the last word before asking for the
reset, so the harness's silent-writer scan could not fire. One mechanism per
premise: the guest establishes it, and the harness judges the kernel's own
count, which the guest cannot write — a boot whose writers never ran stops a
handful of threads and would otherwise pass every judge over a machine that had
nothing to stop. `SPIN_UP` was 30 s against a host that waits 20 s, so the
give-up line it exists to print could not be read; it is 5 s, inside that wait.

Prose the review refused, deleted: the static's restatement of `Stage`'s
contract at a block-layer static, `OpenOperation`'s narration of the two fields
under it, `in_flight`'s argument for its own design, the writers' account of
what the earlier implementation did, and the fourth spelling of the stage split
in the harness.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Three files conflicted, and every hunk of both sides is accounted for.

`kernel/src/arch/syscall/machine.rs`: main added
`panic_console::log_census()` to the closing censuses and this branch added
`log!("{stopped}")` under them. Both survive, the census with its
neighbours and the stop record after all three.

`tests/common/metal.rs`: main added `Readback::panel` and the two priced
panel fields to the pricing loop; this branch added `park_open_operations`
to the same loop and replaced `field.ends_with("_lateness_ms")` with the
named `PATH_TAKEN` list. Both sets of fields are in the loop, and
`PATH_TAKEN` is an explicit three-name list rather than the widening
`8d5956f6` refused — the panel fields are not in it and are priced on every
boot, so a boot that reports none still reds. The comment above the loop is
this branch's trimmed one: the invariant it used to carry lives at
`PATH_TAKEN`'s own declaration.

`tests/metal-profile.toml`: both sides appended a block at the end, so both
blocks are kept, main's panel rows first. Four rows carry a `measured` both
sides moved — `boot.deadlinewedge.complete_ms`, `.back_secs`,
`.deadline_lateness_ms`, `boot.hardlockup.complete_ms` and `.back_secs`.
Main's are run 44's (`fc49eb5b`), this branch's are run 46's (`543a6329`),
and `measured` is the last reading the machine gave, so run 46's stand.
Main's run-44 `panel_max_us`/`panel_us` readings on the same two boots are
untouched by this branch and stay: those two boots now carry one run's
readings for the bounds and another's for the panel, which the next T14 run
over these boots resolves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`ci` run 34896137159 on pull request #452 measured
`quiesce_stops_the_machine` at 4,434 ms across twelve shards — inside
`FAST_COMMIT_MS` with margin, so the name stays `Tier::Fast`. That one row
takes the run's price and only that row: replacing the whole merged artifact
would re-price two hundred names this change does not own, which is what
`47b794b9` was written about. The row also moves to where the merge writes
it, after `query_pci_agreement`.

`sys_shutdown` said "Does not return." and now can: a second caller of the
shutdown is refused and gets `AlreadyExists` back. It says what `sys_reboot`
beside it already said.

`log::user`'s header said "No per-reader state", which `LOG_HOLDERS` is. The
read still has none, and the header now names the table and says why it is
in this file: `SYS_LOG_READ` is the one place in the kernel `Rights::LOG` is
checked.

`quiesce_writers`'s header said the harness matches `WRITING` by its
spelling. It no longer does — the order judge reds on any line under the
boot's last word, whoever wrote it. `WRITERS` is still spelt in both places.

`usb_reset_hands_devices_back`'s deadline arm said it is armed because "the
runner's loop can spawn another job into that gap". On
`tests/jobdeadlinecase` the runner holds no `logread`, so this branch's
first stage bands it before `sync_all` and that spawn cannot happen. The
window is still real and still needs arming under QEMU: it is where the
carved-out log writer is still putting bytes on the volume.

And two the fifth review named that the tree still said: the chronology in
`metal.rs`'s pricing loop, and `metal-profile.toml`'s restatement of the
argument its own `ceiling_from` carries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Seven commits, up to dc38a05 (the ack-delay actuator measures the width of a
shootdown's target set). No conflict: `git merge-tree --write-tree` was clean
and the merge touched no file this branch changed by hand.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
…ling init nobody

`quiesce::claim_the_shutdown` refused a second caller by name with nothing
behind it: deleting it built and left every judge green, because no committed
config issued two shutdowns. Now one does. `tests/quiescetwicecase` runs
`quiesce_twice`, whose child closes a file and asks for the reset while the
new `quiesce-drain-refuse` actuator keeps that file's flush owed — refused on
`iod` every time, so the shutdown's own `writeback::drain_all` is what reaches
it, and refused six times on the thread running the shutdown, so that thread
parks in `block::between_attempts` for 10, 20, 40, 80 and 160 ms between its
attempts. The job, a holder of the log capability and so still running in the
stop's first stage, reads the kernel's log until the first of those refusals
is in it and asks for the reset from the other CPU. The judge,
`quiesce_refuses_a_second_shutdown`, reads where the kernel's refusal lands:
once, between the first and the last of the actuator's lines, with one
`Syncing filesystems...` on the boot, the second caller reporting
`AlreadyExists` from Ring 3, and `Rebooting.` written by the first. With the
guard deleted the second caller runs a second shutdown over the first — two
syncs, no refusal — and the judge names that.

`log::user`'s empty slot was `0`, which is `/system/bin/init`'s pid: the table
issues from zero, init holds `Rights::LOG`, so `holds_the_log(0)` was false by
construction and `note_log_holder(0)` a silent no-op. The empty slot is
`u32::MAX`, the pid no table issues and the one `percpu` spells idle with, and
both functions now refuse it by name. `quiesce`'s caller pair starts there too.

`forget_log_holder` said a pid is issued again. It is not — `ProcessTable` is
an `IdMap` and never reuses a key. The reason it exists is that the table has
eight slots and every job a test boot spawns gets a `SysCap` dup, so the dead
would fill it; that is what the site says now.

`note_log_holder` loads before it exchanges: the steady state is `logd`
finding itself in slot 0 on every read, which is a load and not a `lock
cmpxchg` on the common `SYS_LOG_READ` path.

`quiesce_stops_the_machine`'s workload judge asked for at least six threads and
was met by three writers plus init, logd and test-runner. It asks for exactly
nine — six writers plus those three — so a guest running three writers, or one
that lost a writer to an I/O error before the reset, is the machine the record
says it is not. A writer that fails says which call failed before it goes.

The second stage's `in_flight` clause could not fire: the counter holds only
operations of threads the first stage stops, and such a thread is inside a
device with preemption off, which the stop counts as running first. Deleted.

The check-then-act between `block::counted()` and a sibling thread's first
`SYS_LOG_READ` is written at the site with its bound: a multi-threaded holder
whose first read lands as the stop ends, which no committed config has.

Prose: the count of programs holding the capability, the false "no process"
sentence, the four spellings of the carve-out argument reduced to the one at
`sys_log_read` where the capability is checked, the one-shutdown reason said
once at `claim_the_shutdown`, the "cannot be linked" sentence said once beside
`WRITERS`. The issue file's `metalcase` paragraph was false: `test-runner`
holds the right there but runs no job list, never reads, and is not in the
table — the carve-out on that boot is `logd` alone.

`quiesce_refuses_a_second_shutdown` is registered with the `UNMEASURED`
marker for fast CI to price. `quiesce-drain-refuse` has a `FLASHABLE` ruling
of `Never`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
`quiesce-drain-refuse` kept a closed file's flush owed by refusing it on `iod`
too, and that put the entry in `iod`'s hands for the length of each attempt.
`writeback::drain_retrying` counts the queue under the queue lock alone while
`drain_one` pops under the VFS lock, so a shutdown whose count landed inside
one of `iod`'s attempts counted zero, never parked, and the actuator fired no
times: one boot in five read `refused the shutdown's drain 0 time(s)`, a red
that was the judge's own premise failing and not the kernel's guard.

Under the actuator `iod` now passes over the queue without popping, so the
entry is never in its hands, and the refusals are eight where they were six:
the thread running the shutdown is parked for 10 to 640 ms between attempts,
1270 ms in all, which is the window the second caller has to land in on a
loaded host. Five boots in a row after the change read the refusal between
the first and the last of the actuator's lines; the first of them has it on
cpu1 one millisecond after refusal 1 of 8 on cpu0.

The count-without-the-VFS-lock is `main`'s code and a defect in its own right:
a shutdown can sync and reset while `iod` still owes a budget-refused flush.
Filed with the boot that showed it and not fixed here:
`issues/kernel/the-shutdowns-drain-counts-the-queue-while-iod-holds-an-entry.md`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
…ms it

The previous commit kept `iod` off the write-back queue with a hunk of its own
in `writeback::drain_one`. `main` has had that actuator all along:
`writeback-stall` parks `iod` before it drains, which is exactly "a closed
file's flush is the shutdown's own drain's to find". The hunk is deleted, the
boot arms both names, and `quiesce-drain-refuse` is what it says again — eight
refusals of the mirror write on the thread running the shutdown and nothing
else. Five boots in a row read the kernel's refusal one line-group after
refusal 1 of 8, as before.

The judge counts the shutdowns first. With the guard deleted the second caller
either bands the first where it is parked, or — arriving while the first is
still running — interleaves with it, its stop giving up after 2019 ms on the
one thread that will never reach a safe point and its `Syncing filesystems...`
landing under the first caller's `Rebooting.`. Both shapes write that line
twice, so both are now named as what they are: two shutdowns.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
…e checker says so

Round 7's reviewer named a hypothesis and the orchestrator ordered it run: a
thread `stop_if_blocked` marks while it is parked in `block::between_attempts`
never finishes its syscall, so whatever that syscall had half written stays
half written at the reset.

It is true. `quiesce_leaves_the_volume_whole` stages it with a third actuator
of `fat-mirror-write-refuse`'s shape, `quiesce-fsync-refuse`: from the moment
the machine is stopping it refuses the *active* FAT's write in nine attempts of
a `SYS_FSYNC` flush, after `set_fat_entry` has written the mirror. The caller
is `/system/bin/logd`; its ladder (parks of 10 ms doubling, 2550 ms over nine
refusals) outlasts `log::SHUTDOWN_DURABLE`, which a const assertion holds, so
the stop's second stage meets it parked over two FATs that disagree, with every
other userland thread already stopped. The second stage on purpose: `/log` has
512-byte clusters, so in the first stage `logd` itself would allocate the split
cluster within a few records and heal it by accident.

With the whole fix reverted (three files; M7-quiesce-8.diff) — BUILD EXIT=0,
TEST EXIT=1, the same failure both times, from `toyos-fat32-check`:

    FAT 1 differs from FAT 0 at entry 44: 0x0FFFFFFF against 0x00000000

The fix is the unit the ruling names. `toyos_sched::task` carries `MID_UPDATE`
in the state word, so `stop_if_blocked` refuses a task parked inside an update
in the same exchange that reads `Blocked`; `block::begin_update` is the guard
and `ops::fsync` holds it across its ladder, the one retry ladder a userland
thread parks in. The sweep counts such a thread as running, so the stop waits
for it inside `PARK` and the record's shortfall clause speaks if it cannot.
With it — BUILD EXIT=0, TEST EXIT=0, the checker silent, and the kernel's own
lines:

    [kernel 2.438 cpu0] shutdown: /log did not answer in 2000ms ...
    [kernel 3.006 cpu0] fsync: /log/2026-09-18-151803.log durable on attempt 10 after 2567ms

The judge asserts the harm first, then that the arm fired nine times, that the
shutdown gave up inside that ladder and that the flush closed after it — a
whole volume that does not say so was healed by something else. The price is
that `fsync:` line, a kernel record under the last word on that boot;
issues/kernel/nothing-bounds-the-log-writer-below-the-boots-last-word.md
carries it.

From the same review:

* `forget_log_holder` and its call are deleted. Its reason was false of this
  kernel (a dup records nothing; a holder is recorded at its first read) and it
  had no judge. A slot is never given back; no pid is issued twice.
* The one-shutdown claim moves to `kernel/src/quiesce/claim.rs`, which
  `kernel-loom` compiles: `shutdown_claim` EXIT=0, and EXIT=101 under
  `--features shutdown-claim-split` (a load, then a store) with `two callers
  ran the shutdown`. `quiesce_twice`'s second caller now asks through
  `SYS_SHUTDOWN`, so the one boot judges the claim on both power syscalls:
  the claim moved into `sys_reboot` alone — BUILD EXIT=0, TEST EXIT=1, `this
  boot ran 2 shutdowns`, both times; the guard deleted whole, the same. The
  judge now counts the shutdowns before it asks how the machine ended, since a
  second caller let in through `SYS_SHUTDOWN` may be the one that ends it.
* `Refused::ShutdownDrain.of` and the unused `Clone, Copy` go.
* The `counted()` check-then-act is
  issues/kernel/the-stops-in-flight-count-decides-who-is-counted-at-the-open.md,
  with the true bound: `test-runner` is a multi-threaded holder on
  `tests/testcases`; what no config arranges is its first read beside a
  sibling's operation as the stop ends.
* The workload judge says what its three are: `init` and `test-runner`'s main
  and deadline threads; `logd` is carved out uncounted.
* Prose by deletion: the 1270 ms sentence, "the pid no table issues", and the
  two-callers scenario, which the judge's doc now tells alone.

Also re-measured on this tree: the whole change reverted onto origin/main's
kernel and scheduler (21 files, judges touched 0) — BUILD EXIT=0, TEST
`quiesce_stops_the_machine` EXIT=1, `5 line(s) reached the console after the
boot's last word`; the holder call never taken — BUILD EXIT=0, TEST EXIT=1,
`/log did not answer in 2000ms` under the last word. After the last restore:
BUILD EXIT=0 and all three judges EXIT=0.

Found by reading and filed, not fixed:
issues/filesystem/a-refused-link-write-leaks-the-cluster-append-cluster-just-claimed.md.

`quiesce_refuses_a_second_shutdown` is priced: CI run 35355556050 at 4057307
finished (every guest shard green; `durations` red on the marker, as it must
be), and its `test-durations-merged` artifact reads 4249 ms across twelve
shards. That one row is taken and no other; 4249 is inside `FAST_COMMIT_MS`, so
the name stays Fast and the "Registered UNMEASURED" sentence goes with the
marker. `quiesce_leaves_the_volume_whole` is registered UNMEASURED in its turn.

Gates on this tree, 2026-09-21: the harness's Fast tier 346 passed, 1 failed,
347 total, 412.3 s, EXIT=1 — `handle_kill_policy`, `16 more killed processes
left more live objects behind: [("SharedMem", 5, 6)]`, green alone, on a host
another agent's work had come back to (the same tier took 147.3 s at 4057307);
`--known-red` answers NOT KNOWN-RED, the redlist holding that sentence as a
sighting retired 2026-09-04. `cargo test --lib` EXIT=101 on
`redlist::tests::every_row_can_say_what_it_claims` alone: four of main's rows
measured 2026-08-20/21 are now more than 31 days old; this branch does not
touch `src/redlist.rs`. Workspace host suites EXIT=0, five clippy invocations
EXIT=0 each. The loom control is wired into `.github/workflows/host-tests.yml`,
which `every_model_control_is_wired_into_host_tests` demanded by name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014iqcj4jDKpaiDX8B7CMvmK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant