server: preemption notices, asynchronous parks and exact concurrency together

[danielhanchen]

Stacked on #184. This branch merges #190 (park notices on the stream), #192 (park and restore copies off the decode loop) and #194 (opt-in exact concurrency) so they can be tested together, plus the fixes that only showed once they shared a binary. It is rebuilt from the three branches' heads as they move.

Correction, 2026-09-07. The sentence that used to stand here said every fix found on this branch had been carried back to the PR it belongs to. That is no longer true and it should not have been left standing. git log --no-merges against the three merged heads returns 16 commits unique to this branch, nine of them from 07 September: f8fdf06, 951a219, 57efd21, 7e424ad, 7549c68, 4cf9baf, 00855c2, bb510f5, 09cecf4. They share the park and resume bookkeeping between the copy paths, count the KV reserve in one pass, build the async state io on the host io, and share the preemption test setup. Each one cuts across #190, #192 and #194 together, so they cannot be split back into those branches cleanly.

That makes this branch the merge target for the three, not a scratch integration of them. #190, #192 and #194 stay open as the reviewable layers, and gh api compare confirms this branch contains all three in full (#190 and #192 report ahead=0; #194 differs only by a merge commit with no unique content). It does not contain #186, which still holds four commits of its own including 296 lines of test_preempt_hybrid.py, so the preemption stack currently has two live heads and #186 also needs another merge from #184, which it is four commits behind.

Rebuilt over 75944db80 from the current heads:

1c5aa4904	Merge feat/server-side-preemption-notify (#190, 5a791e03c)
51eca0837	Merge feat/server-side-preemption-async (#192, 888603d03)
70d49bfac	Merge feat/exact-concurrency (#194, da8556d31)
ef257d7b8	Merge feat/exact-concurrency (#194, c6c3cb671): speculative verify batches stay grouped, sliced column split
abccc6e95	Merge feat/exact-concurrency (#194, 379ca5d42): one launch per expert projection under the knob instead of one per token
a7db6262a	Merge feat/server-side-preemption (#184, 50b617ae4): parked slots come back head of line by park time
3f817d880	Merge feat/server-side-preemption (#184, 8057a74af) and feat/exact-concurrency (#194, f4e45646d): resume-order knob logged at load; equal-width grouping under exact mode
e8ab02c94	Merge feat/server-side-preemption (#184, 2a7e277ab): a parked sequence that cannot fit the pool alone is failed instead of parked for ever; margin waived when nothing is resident
db9558696	Merge feat/exact-concurrency (#194, adeed654b): head widths other than 256, tokens with several sequence ids, and hybrid cross-sequence edits refused by name under exact mode
70b874c33	Merge feat/server-side-preemption-async (#192, 01dfecc5e, carrying #184 84311fd3d and 6dbc4e7b6): when the KV-full retry ladder runs out, resident slots are rewound to the cache boundary and the smallest parked instead of every slot getting the context error; a batch holding a draft is never narrowed through the group
a7e661c14	Merge feat/exact-concurrency (#194, c89d87db5): the same, with the page-rounded margin
1130aed0b	Merge #192 47ef8de54 and #194 6573242a1 (#184 86845c15e): a batch holding a draft is narrowed the old way when there is no budget to park into
b89302b25	Merge feat/server-side-preemption-notify (#190, 5864dae95): a park made as a last resort sends the stream comment the planner sends, so Studio shows the pause
f8ec5665a	Merge feat/server-side-preemption-async (#192, cd1cd4e6d): the lookahead margin counts a slot being restored
5e0f99aed	Merge feat/exact-concurrency (#194, 0c9fc0ed8): every context reports the widest decode step it can build, so the column bound covers direct library users
3fc833954	Merge #190 301f480c0, #192 0303888e8, #194 08e10dabb (#184 662ec2029): the planner and the last resort step aside for a context without memory
7c8a6d515	Merge #190 76baf1a68, #192 cac5ee6c0, #194 67063e476 (#184 a9b712eee): unlimited budget enables the last resort; per-slot draft bound for restores and reservations; shared prompt charged once per family; reused slot charged its prefix; context shift before the planner measures
800e166ad	Merge #190 7fb42b582 and #194 46e7fa742: parked prompts of a multi-prompt stream kept apart; ngram-cache verify width; DFlash refused by name; an explicit bound below the decode width refused at context creation; width replayed to backends
eb428268c	Merge #190 53c59a109, #192 8d74efbb2, #194 b7ace528f (#184 64a5064ef): a resident cycling through context shifts takes turns with a parked head
9391aa71e	Merge #190 356978a26, #192 42e536970, #194 dbd82ca5e (#184 6744b3d9f): a non-causal context with a cache refused at creation under exact mode; one page boundary per waiting prompt slot in the reserve cap; a stream parked before its first token starts with the notice; the rotation park announced; a reused slot trimmed to its shared prefix before it is sized or parked; a round with a context shift waits for every copy in flight; the decode width of every context follows the token figure
645d406dd	Merge #190 3306d4b50, #192 ad3856ae9, #194 d4e3fc8a6 (#184 a1c34dad2), on top of 6af0d4e99: a reused slot's need counts from its shared prefix and its trim is safe for memories without partial removal; a waiting child is charged its own cache; equal-count grouping stays on for recurrent and hybrid memories; a width the explicit column bound cannot cover is refused; a restore landing inside the context-shift wait sends the resume notice
276dd2529	Merge #184 6fb0b91c5: LLAMA_SERVER_PREEMPT_POLICY, a test knob to compare victim choices on one workload (off unless set)
ecdb0f531	Merge #190 7efef20fa, #192 2ba0dcf5f, #194 bf00ac38a (#184 a7a04c2a9): the rotation parks the resident that lets the head in (page-rounded here); soft-capped attention refused at load; width reports serialised and monotonic in the backend; the token figure never lowered
089fedd92	Merge #190 3d179fd0b, #192 02a3e11d4, #194 98fe86dfc (#184 64a3f6e7d): a parked slot, or one whose copy is in flight, survives an aborted round; a rotation counts the head's bytes as leaving; the exact setup runs before any context exists; one lock for the token figure, the sequence count and the width
d72e2a40d	Merge #190 a06419d00, #192 e432b15c2, #194 49c66c5bd (#184 55f04bbd6): the leader measured by what a reused slot keeps; no rotation while a park is copying and an asynchronous rotation park re-examined when it lands; the rotation budget under asynchronous copies and buffers returned over budget; the cache-reuse shift waits for copies in flight; an isolated ubatch takes only sets that finish in it; page-aware margin at every step test
b4a5f18d8	Merge #192 2b4a69122, #194 2f2258dc0: a whole-context restore under exact mode is refused before it can clear the cache; the width is published once construction succeeds; the page table is refused on every backend that ignores it; DSpark refused with DFlash; a transfer that fails part way posts no copies; pageable host memory parks synchronously; no transfer for state that is not on a device; the runway rounding explained
48f7d7c0b	Merge #190 1d528856f, #192 faa3dd3f7, #194 23c9dc2ef (#184 270fdd6f3): test that a parent and child alone in the pool get the context error and the server lives
0611fd99b	Merge #192 a4b62f2be, #194 918a8bf4f: a park buffer that comes back pageable parks synchronously from then on; the graphs after a restore wait for its copies on the device; the exact-mode cache asks the device whether it runs the paged attention for the layer rather than trusting the backend's name
f737e6d49	Merge #192 294d2a912: idle parked RAM is given back when another slot needs to park, so a budget that holds one sequence is not spent for good by the first restore; the copies of a park or restore wait for the compute stream on the device instead of draining the host
bf0f73127	Merge #184 3800ddee9 through #190 a158200de and #192 dffa102e3: a pure recurrent cache is served without preemption; a rotation holds both states at once, so the RAM cap is asked for the resident's state in full and a budget that holds one sequence but not two does not rotate, said once per park
a140e80c8	Merge #184 81eec0bb3 through #190 ce02834c4 and #192 a5749d11b: the copies wait for a fence the context records after every decode, so restores issued in one pass run independently on the device; a started slot's reservation counts from the prefix it keeps (page-rounded here, the step starting from that prefix)
e36a4b6dd	Merge #184 af560907b through #190 5ecdba815 and #192 a56d49e1f: the context shift and the planner run inside the guarded part of the step, so a failure there ends the affected slots rather than the loop
b0fae413a	Merge #184 00b27d2df, #190 4918294ff, #192 8cb58dfdf plus 4b91744 here: what a started slot keeps is decided by the batch builder's rule (no caching, aLoRA cutoff); a parked stream keeps a shorter ping it asked for; staging counted by what the buffer charges; fences after the layout check; transfers only where a park can happen; a decode that fits goes ahead beside a park in flight instead of waiting for it; a slot restoring into the prompt phase counts in the page-boundary cap
0d39b88ba	Merge #184 ad89538d2, #190 801e29ad1, #192 114e230ab: the resume order read on every load; the shift wait runs before the draft is asked for, so the draft cache shifts after the copies have landed; the last transfer takes the fences with it
7fdb58bb3	Merge #184 301caa31d, #190 929809ffa, #192 b1a31b42d: the resume order belongs to the context and the recurrent flag is assigned on every load; the shift wait runs before the decode as well as before the draft; a context freed with live transfers drains and disowns them
1bf851916	Merge #184 3c99fafdf, #190 a33c56397, #192 58c6e3329: the pool is measured by the cells each slot physically holds (page-rounded here), and when nothing fits every started slot is trimmed to the prefix its request keeps before idle slots are cleared, so a resume is not attempted against cells the batch builder has not yet released
474569ca3	Merge #190 90a5094d3 and #194 b4b0f9bd7: a started slot's prompt is validated before it can be parked, so a request the STARTED block would reject is never given a notice ahead of its non-stream error; shorter comments from #194, the conflicting hunks keeping this branch's text
f8fdf063a	Comments only: 1307 added comment lines cut to 391 across 44 files (70 percent), the AST gate confirming no code changed; the [TAG_...] anchors stay
951a21931 to bb510f556	Slimmer: the comment-only edits to the repo's pin tooling and CI workflows that a sibling branch's pass had carried in are reverted, the divergence harness leaves the tree, one preempt_park() serves the four park sites and one pair of helpers the three copy completions, the kv reserve is counted in one pass, the two test files share their server setup and stream readers (29 tests and every assertion unchanged), the async host io derives from the sync one, one-reader wrappers and the target-and-draft transfer repeats go. 1050 lines fewer. Verified at each commit by the suite, at the end by the seq-copy and fragmented-restore ctests on the 4B, the exact-mode soak identical 6 of 6 with forced parks, and test-backend-ops under the mode
09cecf4f0	Comments only, second sweep: 341 to 261 added comment lines, of which about 30 are initializer field labels and per-backend anchors; the AST gate confirms no code changed

The four fixes below now come in with those merges. Two conflicts had to be resolved by intent rather than by text: at the KV-full park site the notice and the new host-memory report both belong, in the order the forced-park site already uses; and #194's synchronous planner meets this branch's asynchronous one, where the resolution keeps the asynchronous margin and its RESTORING reservation and takes #194's compile-time-checked rounding helpers, so the two planners share one set of arithmetic and the synchronous branch of the margin rounds as well.

Summary

Three merges and four fixes. Merging #190 into #192 needed intent rather than text: git placed two of the three notice sites after the asynchronous early-outs, where an asynchronous park or restore would have announced nothing. The rule applied is that a park is announced when the slot stops taking part in a decode, on entering the copy-out state, and a restore when it starts again, when the copy back lands; the keepalive therefore spans all three in-flight states. #194's page allocator and #192's park compose without change: a park removes whole pages, both in-flight states hold live cells, and a restore allocates through the page branch of the slot finder.

Fixes found here

• The preemption planner counted tokens while exact mode hands out 256-cell pages, so four chats filled the pool with nothing parked and every request ended in the old context error in three of three rounds. llama_memory_i::alloc_granularity() now reports the page size and the server rounds its four planner figures by it. After: 4 of 4, two parks per round, the parked chat's bytes unchanged.
• A request with n above one reached a cache assert under exact mode and killed the server. It is refused with a 400 at task build, and the cache logs and returns instead of asserting.
• An asynchronous park issued from the victim loop could leave its cells held while the batch was built, so a decode failing for lack of room would halve the batch down to one without ever waiting for the copy. The loop now waits for the park it just issued, and a failed decode waits for an outstanding park before giving up any batch width. This did not reproduce in six staggered live rounds; the change is inert unless the state is reached.
• The 2 s keepalive had never fired: the result queue notifies one condition variable for every result of every task and the reader's timed wait restarted on each wakeup, so on a busy server the timeout never elapsed. The wait now targets a deadline. Three keepalives across a 10.5 s park after, none before, and the same wait serves --sse-ping.

Results

Qwen3.5-4B, two MTP drafts, -c 8192, four slots, seed 0, temperature 0, forced parks every 64 tokens, all three features on, re-run on the rebuilt branch: 4 of 4 in three rounds, 28 parks and 28 restores per round, seven park and seven resume notices on every stream, and the observed chat byte-identical to its solo run in every round, at the same sha256 3922db5817b706aa... the branch gave before the rebuild. Survivor stall at a park 52 to 67 ms and at a restore 39 to 85 ms with exact mode on, against 76 to 78 and 57 to 64 with it off, on a shared GPU.

Qwen3.6-35B-A3B through Unsloth Studio on an unshared GPU: exact mode loads with Studio's own launch args; a chat's bytes are unchanged by three different neighbours across three rounds and by seven forced parks per stream, while the same cell with the mode off diverges at byte 488; four API chats and eight GUI chats in Chromium and WebKit all completed with zero errors and every one of 21 natural parks lasting up to 96 s resumed with the paused label shown.

Cost

Re-measured on this tree, interleaved: three on/off pairs, four chats, prompts of 937 tokens as the server counts them, 2048 tokens each with ignore_eos, -c 8192, four slots, unforced, medians of three.

	exact on	exact off	ratio
speculation off, four-chat aggregate	135.6 tok/s	149.2 tok/s	0.91
speculation off, four-chat wall	79.6 s	68.2 s	1.17
two MTP drafts, four-chat aggregate, at 70d49bfac	71.6 tok/s	159.1 tok/s	0.45
two MTP drafts, four-chat wall, at 70d49bfac	141.5 s	66.9 s	2.12
two MTP drafts, four-chat aggregate, at ef257d7b8	108.1 tok/s	146.5 tok/s	0.74
two MTP drafts, four-chat wall, at ef257d7b8	99 to 105 s	75 to 81 s	1.3

All six runs finished 4 of 4 with parks and restores and no context errors. The 37 to 42 percent this paragraph used to quote was measured on #194's reviewed head and is gone with speculation off: #194's own fixes stopped one prompt serialising every concurrent decode for the whole prefill and dropped the default column bound from a fixed 16 to the width a decode step can reach. What is left there is the CUDA column policy rather than the page bookkeeping, which #194 measures at 0.79 for the kernel half alone against 0.92 for the whole mode.

The speculative cost at 70d49bfac was the batch splitter, not the column policy: under exact mode it isolated every sequence set with more than one token left as a prompt, and a slot's three-token verify batch is such a set, so each decode step ran the whole graph once per slot. #194's c6c3cb671 isolates by width (llama_set_exact_decode_tokens, one plus the draft length) and slices the column split into batch-of-one-equivalent launches, which is the 0.45 to 0.74 in the table, measured on the #194 tree and carried here by ef257d7b8. What remains is the sliced column policy on a twelve-wide verify batch plus paged attention. One B200 shared with another tenant throughout, so the ratios are the result and the absolute figures are not.

Tests

Server harness 31 of 31 (27 preemption, 4 notify, 22 parametrised cases), the fragmented-restore and sequence-copy ctests on CUDA with the 4B, and test-backend-ops 5234 of 5234 in exact mode.

Follow-up: seven fixes and fewer tests

Nine fixes, each checked against the previous head.

• A park right after a context shift saved the pre-shift cells with the post-shift positions, and the resumed run diverged at the token the shift landed on. A new llama_memory_update() applies the recorded shift on the target and the draft before the planner runs, so every park site sees applied state; the reviewer's harness goes from 26621 of 71864 checkpoint bytes differing to 0, and the token-identity test passes.
• A multi-prompt request whose second prompt did not fit could have its first prompt parked, and the park notice opened a 200 stream ahead of the 400. Every prompt of a request is now validated before any task is queued.
• Asynchronous parking is refused for hybrid models as it already was for pure recurrent ones: a recurrent state does not stay in one row while a copy runs beside the decode. The exact-mode soak on Qwen3.5-4B shows 163 parks with zero asynchronous markers.
• Exact concurrency is refused at load for an M-RoPE model with a projector, since the tokens of one image share a temporal position and the paged pool would give them one cell. Text-only M-RoPE models keep working, which a gate on the rope type alone would have broken.
• /slots reports is_preempted only for a slot whose cells are gone and a new is_transferring for a copy in flight, so a reader counting residency keeps counting a transferring slot.
• When --preempt-ram cannot hold a victim, the park drops the victim's cells and keeps its tokens, sampler and counters, and the resume re-prefills the tokens through the ordinary prompt path and carries on from the already-sampled token. Both the planner and the last resort use it, so the stock all-slot context error is unreachable while another slot can be parked. Two 2000-token streams under --preempt-ram 1 go from both erroring to both finishing; a forced park at 1800 tokens with 1819 re-prefilled produces output bit-identical to the unparked run.
• A multimodal chunk is decoded whole inside one iteration, past the batch cap and the KV-full retry, so the planner now reserves the whole pending run of media chunks and the chunk waits for an in-flight park first. Two concurrent 266-token image prompts on tinygemma3 at -c 400 -b 64 go from failed to process mtmd chunk to both completing after a park.

Fewer tests. The two server test files go from 874 lines and 29 tests to 731 lines and 31 tests (22 parametrised cases plus the nine tests above), with the setup, prompts and stream readers shared and the near-duplicates folded into parametrisations. The survivors were checked against four scratch mutations (no victim ever picked, the pre-fix cell count, the started-slot validation gate removed, the park notice removed); each was caught by the intended test. Two behaviours that had no test before are written rather than kept: the physical cell count of a started slot, and the parked keepalive.

One thing found on the way, not of this PR: a 1900-token prompt processed in 32-token batches after a 2000-token decode hangs the CUDA build with graphs enabled, on master as on this branch, with GGML_CUDA_DISABLE_GRAPHS=1 or a larger batch it completes. The recompute test re-prefills in one batch for that reason.

Verified on the merged tree: the two files 31 of 31 on CUDA with the recurrent, hybrid and M-RoPE models supplied and nothing skipped; the upstream completion, context-shift, vision, basic, stream and kv-keep files 73 passed and 1 skipped with the one pre-existing failure, a wording regex on the tiny model that fails on master too; the CPU-only build serves the recompute park and re-prefills 1910 tokens in 32-token batches; the exact-mode soak on Qwen3.5-4B with two MTP drafts and forced parks every 64 steps identical in 6 of 6 rounds against the solo run at 295 to 300 tok/s aggregate, 164 parks and 182 resumes with zero asynchronous markers and no context error. Live through Unsloth Studio: four GUI tool chats at 8k finished 4 of 4 with 9 parks and 9 resumes, and unsloth run at 16k with two MTP drafts and exact concurrency finished 4 of 4 at 191 tok/s aggregate with 5 parks and 5 resumes, no context error in either.

• A batch with a token shared by several sequences under exact concurrency came back from llama_decode as 1, the code callers retry on, when it can never succeed. It is rejected during batch validation with -1 now, before the memory is asked.
• The same for two tokens of one sequence at one position, which the paged pool would give one cell: the batch allocator reports it and decode() returns -1 under exact concurrency.

Round 23

Finding	Fix
A client asking for exact concurrency had no way to tell a build that runs the mode from one that ignores the variable	/props carries exact_concurrency
A recompute park moved the prompt out of the slot, and the draft was bounded by the empty prompt: 2000 tokens and a whole ngram draft could not fit a 2048-cell pool "even alone", failing a request that fits	The draft is bounded by the tokens the sequence comes back with
A multi-prompt request with an invalid token in a sibling parked behind a running one failed inside a stream that had already opened 200	Every sibling is validated in the preflight, with the size checks
A media chunk refused the recompute park, so with the host budget spent nothing could be parked and the pool overflowing ended every chat	The chunk comes back the way it went in: re-encoded off the task, its cells reserved whole; the last resort takes a generating multimodal slot too
Under exact concurrency a re-prefilled sequence is not byte-identical on CUDA: measured on Qwen3.5-4B, identical with parks into host memory, diverging at the second re-prefill, on CPU identical throughout	Decoded tokens are re-prefilled one per step through the narrow path, and the first re-prefill under the mode is logged as not guaranteed identical; Studio sizes --preempt-ram to the pool
The started-slot regression test waited a fixed 50 ms, which a slow host could miss	It waits for the slot to be processing

Round 24: audit fixes and the exact-concurrency follow-ups

Six fixes from an isolated review of this head, each with a test, plus the three code fixes #194 gained since this branch took it (cherry-picked: the weight placement check at context creation, the allocation-granularity comment, and per-page live counts so a removal that empties nothing leaves the page index alone, with tests/test-exact-pages.cpp).

Finding	Fix
A parked multimodal slot was rewound by positions, and an image occupies fewer positions than tokens, so keep_first dropped or kept the wrong tokens	rewind_to_cache converts with size_up_to_pos, steps back to the start of a media chunk it would cut, and asserts the result; tests/test-server-tokens.cpp builds a 256-token, 16-position image chunk and checks the conversion
A host allocation that failed inside a swap park unwound the park instead of falling back	The same victim is parked by recompute (LLAMA_SERVER_PREEMPT_FAIL_SAVE=N fails the Nth save for the test)
A rotation whose resident could not be swapped within the budget never rotated, so a waiter starved behind an unending resident	A second candidate pass without the budget check parks it by recompute; a recompute rotation waits 30 s rather than 2 s, since a re-prefilling resident rotated every 2 s never caught up
Under exact concurrency a prompt added beside decoders was cut into ubatches a solo prefill would not see	A splittable prompt adds whole n_ubatch chunks unless the rest fits, a chunk a neighbour would cut short is deferred, and a batch too small to hold a decode step plus one ubatch warns once; measured identical ubatch grids alone and beside three decoders
A recompute park was silent, so a client under exact mode could not tell a restore from a re-prefill	: recomputed follows the : resumed of such a resume, llamacpp:preempt_recompute_total in /metrics, n_recompute per /slots entry, and preempt: {parks, recomputes} on every final completion object; documented in the server README
The MTP and EAGLE3 drafts ran to the configured window past the caller's remaining allowance, and the server truncated after the fact	The draft loops stop at min(params.n_max, dp.n_max); with one token of allowance left the unfixed build logged the truncation, the fixed build none, and a 4-token draft got three times faster
A batch too small to hold a whole prompt ubatch beside a decode step warned once and filled the shorter remainder, so a prefill beside a running slot was split into ubatches the same prompt does not get alone, with /props still reporting the mode	common_exact_batch_geometry refuses the geometry at init, naming the batch size that would work (-b 514 for a 512-token ubatch and two slots); the warn path is gone, a prompt that does not fit whole waits for the next iteration; tests/test-exact-geometry and a server test
The --preempt-async help said the slots that keep running do not wait for a park	It says what holds: the copy out overlaps with decoding, a copy back in and a kv-full retry behind a copy out that has not landed wait for it; same row in the server README
A park taken during prompt processing was believed to be announced only with the first chunk	Measured: the server already sends the headers and : preempted at park time on every streaming surface (320 ms before the resume in the test scenario); now stated in the README and pinned by a test parametrised over /completion, chat, responses and messages that the notice arrives before the first data line
The rotation test raced the 2 s parked keepalive against the 2 s rotation trigger and failed about one run in five	The test asks for a 1 s ping through sse_ping_interval; ten consecutive runs pass

Verified on one GPU with Qwen3.5-4B: test_preempt.py and test_preempt_notify.py 39 passed, 5 skipped; test_metrics.py 10 passed; test-server-tokens, test-exact-pages (also under LLAMA_KV_CACHE_DEBUG=1) and test-state-seq-copy pass; -ot attn_q=CPU under LLAMA_EXACT_CONCURRENCY=1 is refused naming blk.0.attn_qkv.weight in CUDA_Host; four exact-mode sequences of 256 greedy tokens with a forced park every 64 are identical to their solo references (12 parks, no recompute). The exact and MTP tests skip unless LLAMA_SERVER_TEST_EXACT_MODEL and LLAMA_SERVER_TEST_MTP_MODEL name a model. Not covered end to end: the multimodal rewind, since the only local VLM has media positions equal to its token counts.

[danielhanchen]

Merged #194's c6c3cb6 as ef257d7. It fixes the speculative cost the Cost section measured: under exact mode the batch splitter treated each slot's three-token verify batch as a prompt and gave it a ubatch of its own, so every decode step ran the graph once per slot. Prompts are now isolated by width and the column split runs in batch-of-one-equivalent slices. Two MTP drafts, four chats, the same full-pool shape: 0.74 of the mode-off aggregate against 0.45 before, byte-identical to the solo run on the 4B and the 35B with and without parks. The 16 server tests pass on the merged binary. The Cost section is updated.

[danielhanchen]

Merged #194's 379ca5d as abccc6e: on mixture-of-experts models the exact mode no longer re-enters every expert projection once per token, so the 35B decodes at the mode-off rate within noise with and without MTP drafts, byte-identical to the solo run with and without parks. The 16 server tests pass on the merged binary.

[danielhanchen]

Merged #184's 50b617a as a7db626: parked slots come back head of line by park time, with LLAMA_SERVER_PREEMPT_RESUME=pass keeping the previous order. Server tests pass on the merged build. Evidence and the simulation table are on #184.

[danielhanchen]

Merged #184's 8057a74 (resume-order knob read and logged at load) as 3f817d8, together with #194's f4e4564 (equal-width grouping under exact mode). Tests pass on the merged build.

[danielhanchen]

Merged the second review round: #184's 2a7e277 as e8ab02c (a parked sequence that cannot fit the pool alone is failed with the context error instead of parked for ever, and the scheduling margin is waived when nothing is resident) and #194's adeed65 as db95586 (head widths other than 256, tokens carrying several sequence ids, and hybrid cross-sequence edits are refused by name under exact mode). The merged binary passes the 17 server tests, including the new near-capacity test, and the 35B GUI drive completed 4 of 4 chats with 9 parks and no errors.

[danielhanchen]

Merged #184's 84311fd and 6dbc4e7 as a7e661c: when the KV-full retry ladder runs out, resident slots are rewound to the cache boundary and the smallest parked instead of every slot getting the context error, and a batch holding a draft is never narrowed through the group. Suite 13 + 4 + 2 passed. Exact-mode accuracy with the planner off (last resort only), 4B and 35B, with and without MTP drafts: identical to solo in every round, 0 context errors.

[danielhanchen]

Merged #184 86845c1 (the draft shortcut only when there is a budget to park into) through #192 47ef8de and #194 6573242 as 1130aed, and #190 5864dae as b89302b: a park made as a last resort now sends the stream comment the planner sends. Found by a 35B GUI drive with the planner off: before, four of four chats finished with nine resumes and no pause shown; after, four of four with the pause shown on every chat, 12 parks and 12 resumes, 0 errors. Suites 13 + 4 + 2 passed.

[danielhanchen]

@codex review