server: --pipeline-groups, run the slots over several contexts of one model

[danielhanchen]

Adds --pipeline-groups N to llama-server: the slots are run over N independent
llama_context objects created from the same model, each with its own batch, its own sampling
and its own decode thread. The model weights, the task queue, the results queue and the HTTP
layer are shared. With N = 1 nothing changes.

The point is a layer split across two machines. With one context the split is a two-stage
pipeline fed one batch at a time, so each stage is idle while the other computes. With two
groups there are two batches in flight.

Result

Two DGX Sparks (GB10, 111.9 Gb/s per rail), Qwen3.8-27B-UD-Q4_K_XL, layer split over one
ggml-rpc-server on the peer, -c 16384 --parallel 32 --cache-ram 0 -fa on -ngl 99 -t 6,
32 concurrent clients, 64 requests, npp 128 / ntg 256, best of two repeats, all cells inside one
clock state (2386 to 2402 MHz local, 2433 to 2463 MHz peer, no thermal cap markers).

device order	groups	tok/s (run a / run b)	TPOT ms	TTFT median s	GPU busy local	GPU busy peer
CUDA0,RPC0	1	94.9 / 92.9	310 / 311	7.4 / 7.6	44 / 47 pc	43 / 41 pc
CUDA0,RPC0	2	75.5 / 76.3	395 / 398	6.3 / 5.8	43 / 42 pc	44 / 47 pc
CUDA0,RPC0	2, sampling pool off	73.9 / 75.3	408 / 404	6.5 / 5.8	44 / 43 pc	45 / 46 pc
RPC0,CUDA0	1	99.7 / 98.1	295 / 301	7.5 / 6.6	46 / 43 pc	47 / 46 pc
RPC0,CUDA0	2	130.4 / 131.3	223 / 220	5.7 / 6.1	76 / 76 pc	79 / 76 pc
RPC0,CUDA0	1, --backend-sampling	94.8 / 94.9	309 / 311	7.5 / 7.4	42 / 42 pc	44 / 43 pc
RPC0,CUDA0	2, --backend-sampling	120.4 / 123.7	233 / 230	8.0 / 6.4	71 / 74 pc	73 / 75 pc

1.32x over one context on the same split, at 76 to 79 percent GPU busy per node against 43 to 47
percent. Two things had to be true at once:

1. The device order matters more than the feature. With -sm layer the devices are filled in
   the order they are listed, so the last device holds the output layer. List the RPC device
   first and the local one last (--device RPC0,CUDA0): the logits are then produced locally,
   which removes a n_vocab * n_rows * 4 byte return from every decode step (31.8 MB per step at
   32 rows on this 248320-token vocabulary) and lets the sampler read them out of local memory.
2. Sampling had to come off the critical path. See below.

Slot count sweep, RPC0,CUDA0, --cache-ram 0, -c sized so every slot keeps 512 tokens:

slots	groups 1	groups 2	ratio
32	99.7 tok/s	130.4 tok/s	1.31x
64	117.2	157.2	1.34x
128	124.2	170.1	1.37x
256	124.1 (1 request in error)	130.5 (126 of 512 in error)	not comparable, see limits

What the cost was, and how it was found

LLAMA_SERVER_PIPE_PROF=1 (added here) prints, per group and every five seconds, how long an
iteration spends waiting for the engine, in pre_decode, in llama_decode, in
llama_synchronize and in post_decode, and splits post_decode per token into sampling,
detokenization, stop-string handling and the result queue. On the pair, per group iteration:

	groups 1, 32 rows	groups 2, 16 rows per group
wait for the engine lock	0.00 ms	0.00 ms
pre_decode	0.01 ms	0.01 ms
llama_decode	145 ms	200 to 262 ms
llama_synchronize	148 ms	95 to 126 ms
re-take the engine lock	0.00 ms	0.00 ms
post_decode	21 ms	54 to 76 ms
whole iteration	316 ms	401 ms
per token: common_sampler_sample	0.60 ms	3.6 to 4.6 ms
per token: common_token_to_piece	0.002 ms	0.002 ms
per token: process_token (stop strings, streaming)	0.065 ms	0.18 ms
per token: queue_results.send	0.061 ms	0.18 ms

Sampling one row builds a candidate array over a 248320-token vocabulary, about 4 MB of memory
traffic. With one context that pass runs in the gap where both GPUs are idle and gets the full
memory bandwidth of the node. With two groups it runs against the other group's GPU work on the
same LPDDR5X and costs six to seven times more per row, while sitting on each group's critical
path between the synchronize and the next submit. Every other candidate was measured and
falsified: the results queue and the task queue are three orders of magnitude too small, and the
engine lock was never contended (0.00 ms in every window; forcing the old single shared host lock
back on with an A/B switch gave 74.7 against 73.5 to 75.3 tok/s).

Changes

• --pipeline-groups N, parsed in tools/server because it only means anything for the server.
  Slots are partitioned contiguously; each context gets n_ctx / N and n_seq_max = P / N, so
  the per-slot context and the total KV over all groups are what the user asked for. Slot
  selection still runs over all slots, so prompt-cache similarity and the slot save / restore
  endpoints behave exactly as before. N > 1 is refused together with speculative decoding,
  multimodal, control vectors and --sleep-idle.
• Each group samples its rows over a small worker pool. The rows are independent (own sampler,
  own row of the logits), so the tokens are the ones the serial pass would have produced. The
  thread budget is divided by the number of groups, so a pipeline-groups run is not simply given
  more CPU than the single-context run. LLAMA_SERVER_SAMPLE_THREADS=1 turns it off.
• The engine lock is per group, so the host path of one group does not exclude the other's. This
  is worth nothing on its own on this hardware and is reported as such, but it is what the
  feature is supposed to guarantee.
• Task processing pauses only the group that owns the slot the task touches, instead of every
  group. This is what fixed TTFT (29 s to 7 s at 32 concurrent).
• get_available_slot() called prompt_save / prompt_load, which read and write the slot's
  sequence KV, before the guard waited for the owning group's decode to finish. With
  --cache-ram 0 the cache is null so it never fired; with the cache on it is a live race
  against a running decode and it matches the 29 s TTFT and the abort seen while developing this.
  The cache update now happens after the wait.
• server_metrics is no longer written from several group threads at once, and each group counts
  its own slots instead of every slot of the server (which double counted n_busy_slots).
• ggml-rpc: a whole message is now atomic on the wire. The socket is cached per endpoint and
  shared by every backend of that endpoint, and a message was three unlocked writes, so two
  contexts interleaved their command streams and --pipeline-groups 2 aborted with "Remote RPC
  server crashed or returned malformed response" within seconds. Responses are handed out in
  request order by a ticket, so a waiter does not hold the send lock. last_graph_uid moved to
  the connection and is checked under that lock, which closes a hazard where GRAPH_RECOMPUTE
  could re-run the other context's graph.

Correctness

Per-group speculative decoding (the section below), on top of the proofs already listed here:

• Single GPU, five prompts, 48 tokens, temperature 0: greedy output byte identical between the PR
  base, --pipeline-groups 1 and --pipeline-groups 2 on this branch, md5
  e154ffeace8e6d57298e1963f16529b5 for all three.
• Qwen3.5-4B-MTP on one GPU, --parallel 8: groups 1 and groups 2 give the same sequential greedy
  md5 with MTP (4dae418e8227bddb0fc0e93ab11a9e67) and the same without it
  (43e68a9c6ffafce623dd8089192cdfc3); all eight slots of both groups serve requests and
  /metrics reports drafted and accepted tokens in both MTP arms.
• tools/server/tests/unit/test_speculative.py (--model-draft sidecar, context shift,
  context-not-exceeded, parallel requests): 6 passed with one group, 6 passed with two.
  test_basic.py, test_completion.py, test_ctx_shift.py, test_slot_save.py: 59 passed,
  1 skipped, 1 failed in both arms, the failure being a preset whose model is not in the offline
  cache. Slot save / restore and context shift therefore also run over two groups.
• Greedy output byte-identical at N = 1, N = 2 and N = 4 over a CPU-only two-RPC split,
  five prompts, md5 177dc61e0703eba3bdaf7bf1131f0458, same as the unmodified binary.
• tools/server/tests run serially against this build and against the unmodified branch head on
  the same machine: 325 passed, 233 failed, 6 skipped, 9 errors on both, with byte-identical
  failure name sets. The failures are pre-existing in that environment (the cached test models do
  not match), not introduced here.
• --pipeline-groups 2 with the sampling pool on and off gives the same test results.

Speculative decoding per group

--pipeline-groups N now combines with speculative decoding (--spec-type draft-mtp with the MTP
head inside the GGUF, and --model-draft with a sidecar draft model). Each group owns its own
speculative state:

• one draft or MTP llama_context, created by common_speculative_init_from_params against the
  group's target context, so ctx_other and the next-token embedding hooks point at that context;
• one common_speculative, sized for the group's P/N sequences and addressed by the slot's
  sequence id inside the group (slot.seq_id, which is slot.id with one group);
• the draft batches go through the group's draft context on the group's decode thread, like
  llama_decode already does; the two task-queue yields around the drafter are only taken on the
  single-group main-thread path.

Slot save / restore, context checkpoints (the drafter state is stored and restored with the
checkpoint) and the prompt cache behave as before, per slot. With N = 1 the code path is the same
as before, spelled through groups[0]. With --model-draft the sidecar model is loaded once per
group. validate_pipeline_groups no longer refuses drafters; --mmproj, --control-vector and
--sleep-idle-seconds are still refused with N > 1.

Two DGX Sparks, Qwen3.8-27B-UD-Q4_K_XL, layer split over one ggml-rpc-server on the peer
(--rpc peer:50055 --device RPC0,CUDA0 -sm layer), -c 16384 --parallel 32 --cache-ram 0 -fa on -ngl 99 -t 6, real-text prompts, npp 128 / ntg 256, 2 requests per client. MTP is
--spec-type draft-mtp --spec-draft-n-max 3 off the head inside the GGUF. Two passes with the arm
order reversed, because the SM clock decays through a window; the cells below all ran with both
GPUs at 2390 to 2400 MHz and no cap marker (the three cells that were capped or straddled a cap
stage are listed after the table, and each of them has a clean repeat).

groups	MTP	8 users tok/s	TPOT ms	32 users tok/s	TPOT ms	accepted/drafted 8 / 32
1	no	55.3	136	96.9	311	-
1	yes	95.5 / 93.6	74 / 75	112.9 / 114.2	233 / 232	0.80 / 0.64
2	no	54.0	139	115.7	219	-
2	yes	77.1	84	133.8	200	0.72 / 0.68

At 32 users the combination is the best cell measured: 1.38x over one context without speculation,
1.17x over one context with MTP and 1.16x over two groups without it. At 8 users it is not: one
context with MTP is faster (95.5 against 77.1), because two groups halve the rows per group and the
draft head is at its most valuable when the batch is small and the step is memory bound. The
crossover is between 8 and 32 rows, which is the same crossover the --pipeline-groups result
itself has.

Cells with a clock caveat, each superseded by the clean repeat in the table: one group without MTP
in the first pass ran at 1828 MHz (53.6 / 95.5), two groups without MTP in the reversed pass at
1690 MHz (51.5 / 108.7), two groups with MTP in the reversed pass straddled a cap stage
(92.0 / 128.3). Every 32-user cell in every arm, including the two without speculation, had one
request of 64 return no tokens; it is present in the arms this PR does not touch, so it is not a
property of per-group speculation.

Known limits

• Use --cache-ram 0 with a layer split. The RAM prompt cache moves a whole slot state on
  every slot handover, and on a split most of that state lives on the remote node, so it crosses
  the wire. At 32 concurrent clients on the pair, with the cache at its default: one group 75.9
  tok/s and 33 s median TTFT (against 99.7 and 7.5 with --cache-ram 0); two groups 6.9 tok/s
  with 14 of 64 requests timing out, because the handover runs on the single task thread and the
  other group starves while it does. This is a property of the cache on a split, and one group is
  already badly hurt by it, but two groups make it much worse and it is not fixed here.
• At 256 slots the two-group cell finished 386 of 512 requests, the rest ending with a truncated
  HTTP response; the server log records no error, so the truncation is on the HTTP path at 256
  concurrent streams, and the node crossed 80 C during that cell. That row is reported with its
  error count and is not a clean measurement. 32, 64 and 128 slots are clean.
• The gain is specific to a layer split. On a single node there is nothing to pipeline.

Update 2026-09-07: the flag is worth a good deal more than the figures above

The best number in this description is 1.38x at 32 users with two groups and MTP. That was measured before three launch settings were understood, and with all three the same mechanism reaches 1.77x: 209.5 tok/s against 118.6 for one context on the same split. Qwen3.8-27B UD-Q4_K_XL, two nodes, --device RPC0,CUDA0 -sm layer, both GPUs pinned at 1690 MHz.

rows per group	tok/s	vs one context
32	143.3	1.20x
64	182.7	1.54x
128	209.47 mean (197.2 to 215.8)	1.77x mean, 1.81x best
192	204.3	1.72x
256	206.4	1.73x

What the earlier cells were missing:

• Rows per group. 32 was never swept. The curve is not linear and it peaks at 128, then turns over. This is the whole difference between 1.20x and 1.814x and it is a property of how much work each group has to amortise its half of the step against, not of the flag.
• --kv-unified. Worth 23 to 30 percent at 32 rows on a two group arm and nothing at all on one context, so it never showed up in a single context comparison.
• --tensor-split 0.5,0.5. Without it llama.cpp splits by free memory at load time, so the layer boundary moves between runs and two cells are not comparable. Five boundaries were measured and the even one wins.

Two other things worth recording for anyone reading the numbers above. Speculation does not compose with this at high concurrency: --spec-draft-n-max 3 is the worst of the three depths at every row count on a split, and above 64 rows speculation of any depth loses (212.72 tok/s off against 164.06 at n=1 and 132.24 at n=3 at 128 rows). Acceptance is a property of the depth rather than the rows, 0.87 / 0.78 / 0.69 at depths 1 / 2 / 3, so a deeper draft accepts strictly more tokens per step and still loses throughput. And more groups is not better: four groups is 10.3 percent slower than two and three groups 11.0 percent slower, because the bottleneck GPU is already 91.55 percent busy on CUDA event accounting at two groups and what is left is scheduling inside prompt batches, not idle waiting for a neighbour.

Cost, stated honestly: at 128 rows the TTFT of a saturating burst is 18.1 s median and 27.2 s p99, against 4.6 and 5.6 at 32 rows. --parallel should be sized to the expected load. A server built for 128 rows and driven at 32 gives 110.3 tok/s against 143.3, so oversizing loses 23 percent of throughput and 38 percent of median TTFT at the same time.

Correction 2026-09-08: quote the mean, not the best leg

The 1.814x above was the best of three legs. Re-measured with the original harness on the same binaries, two further legs read 1.789x and 1.665x, with server-side token counters agreeing with the client to 0.01 percent and resident memory matching to the megabyte, so the layer boundary was identical. Pooling all five legs gives a two-group arm of 197.2 to 215.8 tok/s, mean 209.47, against a one-context arm of 118.42 to 118.96, mean 118.63.

So the honest headline for this flag is 1.77x, range 1.67 to 1.81 over five legs, on a saturating 128-row burst whose leg-to-leg spread is 9 percent. The mechanism and the magnitude are unchanged; only the statistic was too flattering.

Two things confirmed while settling this. The comment-reduction pass on this branch is inert: stripping comments with gcc -fpreprocessed -dD -E, all five touched files are token-identical between a1dd7c5e8 and d98d90dd8, and the current head measures 1.759x in the same clock state, inside the leg scatter. And generation length is a real axis worth stating: the same leg reads 1.789x at ntg 256 and 1.714x at ntg 128, because the aggregate ratio is diluted by prefill. The decode-only TPOT ratio is 1.947x.

Scope, relative to the base branch

This PR targets master, and against that base it changes 7 files, 1197 insertions and 231 deletions.
Stated explicitly because a reviewer diffs against the base, not against the point in a stack a
branch was cut from, and a scope claim measured from the wrong reference is exactly the sentence a
reviewer trusts instead of checking.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-09T06:48:36.585451Z	8d97fcc	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8de4320163

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve pipeline groups when spawning router children

When the server starts in router mode, stripping this option before server_models receives argc/argv removes it from the base_preset built in server-models.cpp:407-415; that preset is what gets merged into every child configuration at server-models.cpp:548-552. Consequently, llama-server --pipeline-groups 2 accepts the flag on the router, but each spawned model process starts without it and uses the default of 1, so the new feature silently has no effect in this supported mode. Preserve the setting in the child arguments or otherwise propagate it across the process boundary.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in 55b01c2: the flag is stripped from argv before server_models builds base_preset, so it never reached a child. It is now exported as LLAMA_ARG_PIPELINE_GROUPS, which the child reads when the flag is absent, via common_set_env so Windows is covered.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a1dd7c5e8d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep no-output decodes busy until synchronization

When a prompt is split into batches, intermediate batches commonly have has_output == false, so this skips llama_synchronize() even though llama_decode() is asynchronous. The decode_window destructor then marks the group non-busy, allowing the task thread to acquire its guard and modify that context—for example by launching another slot, updating the prompt cache, or handling a slot operation—while the batch is still executing. Keep the group busy until the asynchronous work is synchronized before exposing the context to task processing.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in 55b01c2: the multi-group path now synchronizes whenever the decode returns 0, not only when has_output. ~decode_window was clearing busy while the decode was still in flight, and every intermediate chunk of a split prompt has has_output false, so this was the common case rather than a corner.

[chatgpt-codex-connector]

Reacquire released group locks before waiting for all

With --pipeline-groups > 1 --cache-idle-slots, the earlier guard.wait_for(slot->id_group) releases every other group's lock, but this subsequent wait_for_all() assumes all entries in lks are still owned and never reacquires them. If another group is busy, it passes an unlocked unique_lock to condition_variable::wait; even when the predicate skips that call, the following cache loop reads and clears the other groups' slots and contexts without excluding their worker threads. This can race or fail as soon as a new task starts while another group is decoding.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in 55b01c2: wait_for_all now releases what is left and retakes every lock in group order before waiting, so it never waits on an unowned unique_lock and the cache_idle_slots loop that walks every group's slots actually holds their locks. The constructor's ordering is kept so two guards still cannot deadlock.

[chatgpt-codex-connector]

Synchronize profiler snapshots across groups

When LLAMA_SERVER_PIPE_PROF=1 and multiple groups are active, the reporting group reads and resets every group's plain server_group_prof fields here while the other workers concurrently update those fields, including during their unlocked decode windows. The atomic report timestamp only selects a reporter; it does not protect these counters, so profiling introduces C++ data races and can emit corrupted measurements. Snapshot or reset each group's counters under appropriate synchronization.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in 55b01c2: each group now reports and resets only its own counters, under its own lock, with a per-group window. The shared atomic only elected a reporter, which then read and cleared the other groups' plain counters while their workers were still writing them. Per group costs nothing on the hot path, unlike making the counters atomic.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d98d90dd8f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Avoid concurrent access to the shared llama context

With the default sampling pool enabled and at least two ordinary sampling rows in one group, these workers call common_sampler_sample() concurrently on the same llama_context. The preparatory llama_get_logits_ith() does not make that safe: each sampler call invokes context getters whose llama_context::output_reorder() unconditionally executes output_swaps.clear() (src/llama-context.cpp:2222-2287), so the workers concurrently mutate the same vector and introduce a C++ data race on every such decode. Serialize the context getter phase or provide each worker immutable row data rather than passing the shared context into parallel calls.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in 55b01c2 by removing the parallel sampling pool. The problem is wider than output_swaps: common_sampler_sample opens with llama_synchronize, which does non-atomic read-modify-writes on t_eval_us, n_eval and n_queued_tokens, and set_logits then re-enters the context through six more getters. Making that safe means duplicating a large slice of the context API for an optimisation with no recorded measurement, so the sequential path stands on its own. Re-measuring the row sweep against the 1.77x record is the remaining follow-up.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 55b01c2926

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Attach persistent thread pools to every pipeline context

When --pipeline-groups > 1 is used with CPU or partially offloaded computation, contexts after group 0 are created directly with llama_init_from_model() and never receive the common_threadpools::init() performed for the first context by common_init_from_params(). Their CPU backend therefore creates and destroys a disposable thread pool for every graph execution (ggml_graph_compute()'s null-threadpool fallback), adding substantial per-token overhead and ignoring configured affinity, priority, polling, and strict-CPU settings. Give each additional context a persistent, correctly configured thread pool with a lifetime at least as long as the context.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in be0bbf9. Correct: only group 0's context gets the pool from common_init_from_params(), so later groups fell back to a throwaway pool per graph and silently ignored --cpu-mask, --prio, --poll and --cpu-strict. Silently discarding an explicit user setting is worth fixing even though the cost is small at -ngl 99, since CPU and partially offloaded runs are where those flags actually get set.

The pool is owned by server_group and declared after ctx, so it is destroyed by groups.clear() after the llama_free() above it, on the normal and the error path alike. Verified one pool at 1 group, which is unchanged, and one per group at 2.

[chatgpt-codex-connector]

Remove the nonexistent sampling-pool controls

The final implementation samples rows serially in post_decode() and contains no LLAMA_SERVER_SAMPLE_THREADS lookup or sampling worker pool, so users following this documentation cannot enable, disable, or size the advertised pool and the stated performance explanation no longer describes the shipped code. Either restore a safe implementation or update this section and its benchmark expectations to match the serial sampling path.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in d1d1a12. The pool and LLAMA_SERVER_SAMPLE_THREADS were removed in 55b01c2 and the README was not updated with them, so this is a real mismatch I introduced. The section now describes the serial path and says why the pool was removed.

[chatgpt-codex-connector]

Fit memory for all pipeline contexts

When the default parameter fitter is active near a device's memory limit, dividing n_ctx before common_init_from_params() makes common_fit_params() budget KV and compute memory for only one C/N context. Model placement is then fixed, after which the code allocates N such contexts (and potentially N speculative contexts), so the fitter can select a placement that fits its estimate but fails while creating a later group. Account for every group's context allocations during fitting rather than presenting only one group's cost.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in 0b039f8, and this was the most serious item in the round: on a 121.69 GiB node the failure mode is an out-of-memory rather than a slowdown.

I tried your framing first, fitting the undivided n_ctx, and measured that it is not sufficient. With common_get_device_memory_data on a 1.5B, the KV tracks n_ctx (112, 224 and 448 MiB at 4k, 8k and 16k) but the compute buffer does not (536 MiB flat across that whole range). Fitting the full n_ctx therefore budgets the KV correctly and still leaves n_groups - 1 compute buffers unaccounted for.

So the extra contexts go into the fitter's per-device margin instead, which covers both terms. Weights are not counted for the target model or for an MTP draft, which share them; only a separate --model-draft adds weights per group. Gated on n_groups > 1 and params.fit_params: verified that --pipeline-groups 1 reserves nothing, that -fit off reserves nothing, and that the reservation scales as expected (175 MiB for one extra context at 2 groups, 356 MiB for three at 4 groups, lower per context because n_ctx/N is smaller). The amount is logged per device so it can be audited.

[chatgpt-codex-connector]

Select a group with enough slots for child completions

For an automatically scheduled n > 1 completion, the parent slot is selected globally before checking whether its pipeline group has enough free child slots. If prompt similarity repeatedly selects a cached slot in a partly occupied group, this branch defers the request even when another group has enough idle slots, and each retry selects the same cached slot again until that group's unrelated generation finishes. Filter candidate parent slots by same-group capacity, or retry selection in another eligible group before deferring.

Useful? React with 👍 / 👎.

[danielhanchen]

Not taking this one. It is a scheduling inefficiency rather than a correctness defect: the request is deferred and retried, and it makes progress as soon as the unrelated generation in that group finishes, so nothing is dropped or corrupted. Worth revisiting if it ever shows up as measurable head-of-line blocking.

[chatgpt-codex-connector]

Register the new option in generated CLI help

Because --pipeline-groups is consumed before common_params_parse() and has no registered common_params_option, llama-server --help cannot list this newly supported public flag or its environment variable. Users relying on the executable's canonical option reference therefore cannot discover its syntax or constraints; register a server-scoped parser/help entry while preserving any router propagation needed after parsing.

Useful? React with 👍 / 👎.

[danielhanchen]

Not taking this one. --help is generated from common_params_option, and registering the flag there is what the design deliberately avoids: it must be consumed before common_params_parse, which rejects unknown options, and it changes nothing outside tools/server. The flag and its environment variable are documented in tools/server/README.md, so this is a discoverability gap rather than incorrect behaviour.

[danielhanchen]

Done in 8d97fcc. To be clear about why, since I turned this down in an earlier round: the earlier rejection is superseded by a new argument rather than reversed on the old one.

What I rejected then was the discoverability case, that a user typing --help should see the option. That is still not on its own a reason to move parsing, and I would turn it down again on those terms. What has changed is that a correctness problem surfaced in the pre-scan itself while working the -- bug in the previous commit. Bounding the scan at a bare -- protects operands after the separator, but before the separator a pre-scan has no option table and so cannot tell an option from the value of a preceding option. A literal --pipeline-groups passed as some other option's value was consumed and stripped from argv. That is not a discoverability complaint; it is the pre-scan reading arguments that are not its own, and no amount of care inside a hand-written scan fixes it, because the information needed to fix it is exactly the option table the scan runs before.

So the option is now registered normally. n_pipeline_groups joins common_params, the option is declared for the server example with set_env("LLAMA_ARG_PIPELINE_GROUPS"), and the >= 1 check moved into the parser, where a bad value reports through the usual error path instead of exit(1):

--pipeline-groups N       run the server slots over N independent contexts of one model, so one
                          group decodes while another is between steps (default: 1)
                          (env: LLAMA_ARG_PIPELINE_GROUPS)

$ llama-server --pipeline-groups 0
error while handling argument "--pipeline-groups": error: --pipeline-groups must be >= 1

The environment path matters beyond convenience: it is how router mode reaches its children, which are spawned rather than given a command line, and registering the option gets that documented in --help rather than only in the README. The README paragraph describing the pre-scan limitation is gone, since the limitation is gone with it.

[chatgpt-codex-connector]

Restrict failed parent-task cleanup to the locked group

When a multi-completion launch fails after this wait—for example because a child has an invalid sampler or aLoRA configuration—launch_slots_with_parent_task() invokes release_slots(), which scans every server slot and reads state and task. wait_for() has released every other group's lock, so their workers can mutate those fields concurrently, creating a C++ data race while handling an otherwise recoverable invalid request. Since all parent and child slots are now guaranteed to belong to one group, limit cleanup to that group's slots or retain the necessary locks.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in d1d1a12. Correct, and it is the same family as the wait_for_all item from the last round: that fix covered wait_for_all but not this scan. Since the parent and its children are guaranteed to share a group, release_slots now iterates only that group's slots.

[danielhanchen]

Review round: what was broken, what changed, and what it cannot break

Five items from the first round and two from the second are fixed. Each one below is stated as the
concrete behaviour before, the behaviour after, and what it means for the paths that must not
regress. --pipeline-groups lives in llama-server and is not behind the RPC gate, so every
one of these is live for someone running a single GPU with no peer and no interconnect. That is the
lens used throughout.

1. Router children silently ran with one group

Before: server_take_pipeline_groups() compacts argv before llama_server(params, argc, argv)
reaches server_models, whose base_preset(ctx_preset.load_from_args(argc, argv)) is merged into
every child preset. llama-server --pipeline-groups 2 was accepted on the router and every spawned
child started with the default of 1. The feature was silently absent in a supported mode.
After: the router pushes LLAMA_ARG_PIPELINE_GROUPS onto child_env, next to the existing
LLAMA_SERVER_ROUTER_PORT, and the child reads it when the flag is absent.
Real: yes. A user-specified flag was accepted and discarded.
Breaks nothing: the child_env entry is only added when n > 1, so a single-group router spawns
byte-identical children.

This went through a wrong version first. It originally wrote the variable with common_set_env.
That is not safe on Windows: server_models builds child_env from get_environment(), which reads
the Win32 block via GetEnvironmentStringsW, while common_set_env writes the CRT copy via
_putenv_s, and CRT to Win32 propagation is not documented
(_putenv_s,
Changing Environment Variables,
curl#4774 for the same divergence in the other
direction). The value could have been invisible to exactly the code that needed it, on Windows only.
Pushing onto child_env removes the question entirely and needs no platform-specific code.

Simulated on Linux, one axis per run, each verified by whether extra contexts were created:

flag	env	groups
absent	absent	1
absent	2	2
absent	`` (empty)	1
--pipeline-groups 2	absent	2
--pipeline-groups=2	absent	2
--pipeline-groups 1	absent	1
--pipeline-groups 1	2	1 (the flag wins)

The empty-string row matters for portability: _putenv_s(name, "") deletes the variable on
Windows rather than setting it empty, so getenv there returns NULL. common_get_env maps NULL to
"" and the code skips on empty, so Windows and Linux land on the same behaviour by two different
routes. I cannot build or run Windows here; that part is read from the documentation above rather
than executed, and it is the one claim in this PR that is reasoned rather than measured.

2. A decode was left in flight while the group advertised itself idle

Before: llama_decode is asynchronous. With has_output == false the synchronise was skipped,
then ~decode_window relocked, cleared busy and notified. A task thread could take the guard and
touch the context while the GPU was still executing. Every intermediate chunk of a split prompt has
has_output == false, so this was the ordinary path for any long prompt, not a corner.
After: the multi-group branch synchronises whenever the decode returns 0.
Real: yes, and it is the most likely of these to have produced the rare corruption reports.
Breaks nothing: the change is inside if (n_groups > 1). The single-group branch still reads
if (ret == 0 && has_output) and is unchanged, so the default configuration, which is what a
single-GPU user runs, is bit-for-bit as before.

3. wait_for_all() waited on locks it did not hold

Before: wait_for(id_group) calls release(g) on every other group, which unlocks. A later
wait_for_all() on the same guard passed those unowned unique_locks to
condition_variable::wait, which is undefined, and the --cache-idle-slots loop that follows walks
every group's slots calling prompt_save, prompt_cache->update and prompt_clear without
holding their locks. Reachable with --pipeline-groups > 1 --cache-idle-slots.
After: it releases what is left and retakes every lock in group order before waiting.
Breaks nothing: the constructor's group ordering is preserved, so two guards still cannot
deadlock, and with one group the guard is a no-op as before.

4. Failed parent-task cleanup scanned every group's slots

Before: release_slots() iterated the global slot vector reading state and task while
wait_for() had released the other groups' locks. Same family as item 3; the first fix covered
wait_for_all but not this scan.
After: the parent and its children are guaranteed to share a group, so it iterates only that
group's slots.

5. Profiler counters were read and reset across groups

Before: the atomic only elected a reporter, which then read and cleared every group's plain
int64_t counters while their workers were writing them.
After: each group reports and resets only its own, under its own lock, with its own window.
Breaks nothing: LLAMA_SERVER_PIPE_PROF is off by default and making the counters atomic
instead would have cost the hot path.

6. The parallel sampling pool re-entered a shared context

Before: workers called common_sampler_sample() concurrently on one llama_context. The
review cited output_swaps.clear(), which is real but bounded, since the hoisted
llama_get_logits_ith leaves the vector empty. The wider problem is that common_sampler_sample
opens with llama_synchronize, which does non-atomic read-modify-writes on t_eval_us,
t_p_eval_us, n_eval, n_p_eval, n_queued_tokens and t_compute_start_us and calls
ggml_backend_sched_synchronize, and set_logits then re-enters through six more getters.
After: the pool is removed. The sequential path it fed was already complete and correct.
Breaks nothing, but this is the one that needs measurement rather than argument. It changes the
single-node path too, and a regression there would matter more than the split figure. That
measurement is running and I will post it when it lands, including a one-context single-Spark leg
with no RPC at all.

Still open

Two items from the second round are not triaged yet and I have not reacted to them: fitting
memory for all pipeline contexts, and attaching persistent thread pools to contexts after group 0.
The first looks genuine and matters for the "never OOM" rule, since common_fit_params() sees one
context's cost and N are allocated after placement is fixed. Neither has been dismissed.

None of these changes key on a model, a family or a checkpoint.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 70f76359ef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Replace inherited pipeline-group values for router children

When the router inherits LLAMA_ARG_PIPELINE_GROUPS=2 but is explicitly started with --pipeline-groups 1, this condition adds nothing, so base_env still passes 2 to every child and the CLI override is lost. Conflicting values above 1 are also appended as duplicate environment entries; on POSIX, getenv() returns the first inherited entry. Fresh evidence in the propagation fix is that child_env copies base_env unchanged before conditionally appending the resolved value. Remove any inherited entry and always insert the router's resolved setting.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in c790aa4. Confirmed both directions you describe. base_env is a copy of the router process environment (get_environment over environ), so appending could not override an inherited entry, since execve keeps duplicates and getenv returns the first, and an explicit --pipeline-groups 1 appended nothing at all. The inherited entry is now erased and the resolved value always set, including 1.

[danielhanchen]

The concurrency fixes, under ThreadSanitizer

Argument is not evidence for a race, and neither is one clean run. These were checked with TSan,
which flags the race whether or not the window is hit, so a flip pair is meaningful rather than
lucky.

Two builds, d98d90dd8 (before) and the current head (after), both
-DGGML_CUDA=OFF -DGGML_RPC=OFF -DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLAMA_SANITIZE_THREAD=ON.
Identical workload against each: --pipeline-groups 2 --cache-idle-slots --parallel 8, -b 32
so prompts split into many batches and most carry has_output == false, eight completion workers,
two n_cmpl workers including requests deliberately over the per-group slot limit so the launch
fails and release_slots() runs, and three workers polling /slots and /metrics continuously so
the task thread keeps taking the guard while a group decodes. Three independent runs of each.

run	before	after
1	10 warnings, 2 in the server slot paths	8 warnings, 0
2	9 warnings, 13 slot-path frames	7 warnings, 0
3	9 warnings, 13 slot-path frames	7 warnings, 0

The two that disappear are exactly the fixed code:

WARNING: ThreadSanitizer: data race
  server_context_impl::pre_decode(server_group&)::{lambda(server_slot&)#5}::operator()
  server_slot::is_processing()

WARNING: ThreadSanitizer: data race
  server_slot::is_processing()
  server_slot::release()

One group's decode thread writing a slot while the task thread reads is_processing(), and a slot
being released under a concurrent read. That is the wait_for_all and release_slots family, and
it is gone in all three runs after the fix.

The remaining warnings are identical in both builds and are all in src/llama-* or are
memcpy under posix_memalign/pthread_create from the ggml CPU backend, whose spin-barrier
synchronisation TSan cannot see. They are not introduced by this PR and they do not move, so they
do not confound the comparison.

What this run does not prove. It is -ngl 0, so llama_decode is not the asynchronous GPU
submit that the ~decode_window fix is about. The flip above is evidence for the slot-state fixes,
not for the synchronisation one. That fix rests on the source argument instead: the sync is inside
if (n_groups > 1), so the single-group default is unchanged, and busy can no longer be cleared
with work outstanding. TSan cannot instrument the CUDA stream, so an equivalent flip is not
available for it.

Reproducer: tsan_stress.sh in the working notes. On this kernel TSan needs
vm.mmap_rnd_bits=28 or setarch -R, otherwise it aborts with "unexpected memory mapping".

The two remaining items: both genuine, neither fixed

Fit memory for all pipeline contexts. Confirmed. params_ctx.n_ctx = params_base.n_ctx / n_groups
runs before common_init_from_params(), so common_fit_params() budgets one context of C/N and
fixes the model placement from that estimate. N such contexts are then created with
llama_init_from_model() after placement is already decided, so the fitter's headroom is short by
N-1 contexts' worth of KV and compute, and a placement that fits on paper can fail when a later
group's context is created. It is gated on params.fit_params, so it needs auto-fitting, which is
the default when placement is not pinned.

I have not fixed this and have not reacted to it. There are two candidate fixes: raise
params.fit_params_target, the per-device margin, by N-1 contexts measured with
common_get_device_memory_data(), which costs an extra probe at startup; or run the fit against the
full n_ctx and n_parallel and only then create the divided contexts, which needs the fitted
placement copied back before the second call. Both change how memory is fitted, which is the one
path where a hasty change is worse than the defect, so it should be its own change with its own
near-limit memory testing rather than a rushed addition here.

Persistent thread pools for contexts after group 0. Also confirmed.
pimpl->threadpools.init(lctx, params) in common_init_from_params runs for group 0's context
only; groups created with llama_init_from_model() never get one, so their CPU backend falls back
to a disposable per-graph pool and silently ignores the configured affinity, priority, poll and
strict-CPU settings. Real, though the impact is small with -ngl 99 and concentrated on CPU or
partially offloaded runs. Also not fixed and not reacted to; it needs per-group pool ownership with
a lifetime at least as long as the context.

Neither has been dismissed. I would rather leave both visibly open than mark them agreed without a
commit behind them.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be0bbf9791

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Map draft reservations by device identity

When --device-draft orders the devices differently from the target model, this positional update charges each draft allocation to the wrong target-device margin. For example, target devices CUDA0,CUDA1 with draft devices CUDA1,CUDA0 swap the added model/context/compute costs, even though the normal fitter maps its extra model by matching ggml_backend_dev_t; near the memory limit, --fit --pipeline-groups 2 can therefore approve a placement and then fail while creating the second draft model/context. Map each entry in devs back to the target model's device identity before updating fit_params_target.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in 64e05a2. You are right, and the code even said so: my comment claimed the memory data and the margins share a device order, which I asserted rather than checked. The draft reservation now matches on ggml_backend_dev_t the way common_params_fit_impl maps its extra model.

There was a second misindexing in the same loop that this found. common_get_device_memory_data returns one entry per device plus a host entry at the back, while margins are built over the devices alone (common/fit.cpp, margins is filled over nd), so running the loop to mem.size() charged host memory to a device margin. Only devices are charged now, except when there is no device at all, where the single margin is the host one.

[chatgpt-codex-connector]

Attach thread pools to per-group draft contexts

With --pipeline-groups > 1 and draft/MTP speculation that performs CPU work, every newly created draft context here lacks a persistent common_threadpools: common_speculative_init_result owns only the model and context and never calls llama_attach_threadpool. The fresh evidence after fixing target contexts is that only groups[g]->ctx receives a pool at lines 1242-1243, while these additional draft contexts still use disposable fallback pools and ignore draft CPU affinity, priority, polling, and strict-CPU settings.

Useful? React with 👍 / 👎.

[danielhanchen]

Fixed in 7bd64bb. Each group now owns a common_threadpools for its draft context, declared before spec_init so it is destroyed after the context it is attached to.

One detail worth stating: it is initialised from params_dft, not params_ctx, so the draft cpu params are the ones that take effect. This also gives the draft context a pool when there is only one group, which upstream never attached.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c790aa4bb3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Recompute output limits after partitioning the slots

When --pipeline-groups > 1 is used for ordinary generation or speculation, params_ctx.n_parallel is reduced here but n_outputs_max still contains the limit calculated from the server-wide parallelism at the start of load_model(). llama_context::reserve() uses that unchanged ceiling when reserving the prompt-processing graph, so every context reserves compute buffers for up to all P output rows even though its group can produce only P/N rows (or the corresponding per-sequence speculative multiple). With large vocabularies or high --parallel, this needlessly multiplies compute memory and can force --fit to reduce offload or reject a configuration that would fit. Recompute the output limits from params_ctx after dividing n_parallel.

Useful? React with 👍 / 👎.

[danielhanchen]

Accepted, and it fits the pattern of the other two memory items: the group split changes what each context needs but the reservation and the reserve path were both still working from the server-wide numbers.

Not fixed yet and no thumbs-up. Taking it together with the output buffer item, since both change what a single group actually reserves and both should be validated against a real near-limit fit rather than by reading the code. The near-limit test for this PR has not run yet, which is the honest reason none of these three has a commit against it.

[chatgpt-codex-connector]

Reserve each additional context's output buffer

With the default fitter active near the memory limit, this reservation includes only the context and scheduler compute values returned by common_get_device_memory_data(). That API derives its result from llama_context::memory_breakdown(), which does not include the separately allocated buf_output; nevertheless every context created at line 1237 allocates its own output buffer. The omitted cost is proportional to vocabulary size and output rows, and becomes substantially larger with backend sampling, so fitting can approve placement and then fail while allocating a later pipeline context. Include the extra groups' output-buffer cost in the reservation.

Useful? React with 👍 / 👎.

[danielhanchen]

Confirmed. llama_context::memory_breakdown() sums model, memory and scheduler compute only; buf_output is allocated separately in output_reserve() through ggml_backend_buft_alloc_buffer and never appears in the breakdown, so common_get_device_memory_data cannot see it and my reservation does not include it.

Accepted, not fixed yet, so no thumbs-up until there is a commit. Worth noting for whoever picks it up that the near-limit test that would actually demonstrate this has not run yet, so the fix should be validated against a real fit rather than by inspection.

[chatgpt-codex-connector]

Account for draft-only devices when reserving memory

When --device-draft contains a device that the target model does not use, this identity-matching loop finds no devs_tgt entry and silently drops that draft device's reservation. For example, a target on CUDA0 and a sidecar draft on CUDA1 receives no reservation for the N-1 additional draft models and contexts loaded for pipeline groups; if CUDA1 fits one draft but not all of them, --fit still approves the configuration and a later group fails during initialization. Handle draft-only devices explicitly rather than reserving only the intersection of the target and draft device sets.

Useful? React with 👍 / 👎.

[danielhanchen]

Accepted. This is a direct consequence of the identity matching I added in 64e05a2, and I had already noted the intersection-only behaviour without treating it as a defect, which was the wrong call: silently dropping a draft-only device is exactly the failure mode --fit exists to prevent.

One structural point that shapes the fix. fit_params_target is indexed by the target model device list that the fitter builds, so a device the target does not use has no margin entry to charge, and common_params_fit_impl has the same limitation for its extra model. So a fix that claims to budget CUDA1 in your example would have to extend the fitter device list, not just this loop.

Not fixed yet, so no thumbs-up. My intent is to charge it where a margin exists and warn explicitly where none does, so the case stops being silent even where it cannot be budgeted, and to say so rather than imply full coverage.

[chatgpt-codex-connector]

Preserve per-sequence capacity with unified KV

When --kv-unified is enabled, dividing n_ctx by the group count reduces the maximum context available to any individual request from C to C/N: llama_context sets n_ctx_seq = n_ctx for unified KV, rather than dividing it by n_seq_max. This is especially likely with pipeline groups because automatic --parallel enables unified KV, and the README also recommends it for throughput; a long request that fits with one group can therefore be truncated or fail with context exhaustion after enabling two groups. Either retain C for every unified-KV context (and account for the additional memory) or explicitly expose and validate the reduced per-request limit instead of claiming that per-slot context is unchanged.

Useful? React with 👍 / 👎.

[danielhanchen]

Confirmed in the source, and this is the most serious of the four. src/llama-context.cpp sets n_ctx_seq = n_ctx in the unified branch and only divides by n_seq_max in the non-unified one, so dividing n_ctx by the group count does cut the maximum single-request context from C to C/N under --kv-unified.

It also lands squarely on how this is actually being run: the pair measurements for this PR use --kv-unified, so the configuration I have been quoting numbers for is exactly the one where a long request silently loses half its context.

Accepted, not fixed yet, so no thumbs-up from me until there is a commit. Of your two options I intend to take the second, expose and validate the reduced per-request limit, rather than retaining C per context: retaining C would multiply the unified KV allocation by N, which is the opposite of what the fit reservation work in this PR is trying to control. The README claim that per-slot context is unchanged is wrong for unified KV and will be corrected with it.

[danielhanchen]

Following up on my previous reply, because working the arithmetic out properly changed my answer. I said I would take your second option, expose the reduced per-request limit. I now think that is the wrong choice, and I want to say so rather than quietly do something else.

The two KV modes are not merely different here, they are opposites, and that is what makes this a design question rather than a doc fix. This PR divides both n_ctx and n_parallel by the group count. Non-unified then gives n_ctx_seq = (C/N) / (P/N) = C/P, exactly what it was, so the README claim that per-slot context is unchanged is correct in that mode. Unified sets n_ctx_seq = n_ctx directly, so the same division gives C/N and the claim is false. The division is precisely right in one mode and precisely wrong in the other.

That matters because under unified KV, -c is the per-request ceiling: it is the number the user set and the only one they can observe. Halving it is not reducing a limit, it is redefining the flag. Documenting that would make the surprise findable but would still ship a flag whose meaning depends on an unrelated flag.

So: do not divide n_ctx when kv_unified is set. Give every group the full C, add the extra contexts KV to the fit reservation, and let the fitter arbitrate. My earlier objection to this was that it multiplies unified KV by N, which is true and is a real cost, roughly C worth of KV becoming N times C where the divided form stays at C. But that cost is exactly the kind of thing the reservation work in this PR exists to account for, and if it does not fit, the fitter reduces n_ctx and the user sees a smaller C they can act on. A visibly smaller C beats a silently halved one.

Two consequences I will carry with it: the reservation has to include this, which ties it to the output-buffer item you filed alongside, and the README sentence needs to state the per-request ceiling per mode instead of claiming it is unchanged in general. A startup line reporting the per-request context and the total is worth having in both modes regardless of which way this lands.

Flagging one thing I cannot yet support with evidence: the N-times-KV cost under unified is arithmetic, not a measurement, and the near-limit fit test that would show whether the fitter handles it gracefully still has not run. I would rather land this after that test than before it.

[danielhanchen]

Fixed in 8989e08, taking the option I argued for in the follow-up rather than the one I first said: n_ctx is no longer divided when kv_unified is set, so each group is created with the full C.

The split-KV path is unchanged, because there the division is correct: n_ctx_seq = n_ctx / n_seq_max, so dividing n_ctx and n_parallel together leaves the per-request context at C/P. Only the unified path, where n_ctx_seq = n_ctx, was being harmed by it.

Two things came with it. The multiple-of-N requirement on --ctx-size now applies only to the split path, since only that path divides. And the README claim that the per-slot context is unchanged, which was true for split KV and false for unified, now states both modes and says plainly that unified costs N times the KV.

That memory cost is real and is not hidden: reserve_extra_group_memory measures each extra context and charges it to the fit margins, so a configuration that no longer fits comes back as an n_ctx the fitter lowered and reported. I want to be clear that this composition is arithmetic rather than something I have watched happen at a real limit yet; the near-limit fit test is still queued behind other work on the pair, and it is the thing that would confirm the fitter degrades gracefully here rather than failing late.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 50c051472c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Do not promise automatic context shrinking

With --pipeline-groups > 1, this fallback cannot occur: validate_pipeline_groups() requires an explicit nonzero context size (server-context.cpp:1563-1565), while the fitter enables context reduction only when cparams->n_ctx == 0 and explicitly leaves user-set sizes unchanged (common/fit.cpp:197,453-455). If the multiplied KV allocation exceeds memory, fitting may change model placement or loading may fail, but it will not lower and report n_ctx; correct this guidance or implement the advertised fallback.

Useful? React with 👍 / 👎.

[danielhanchen]

Re-measurement 2026-09-09: 1.68x, not 1.77x, on tonight's machines

Re-confirmed from a clean slate at this head (d98d90dd8): fresh worktree fetched as
refs/pull/187/head, fresh build, fresh venv, and the same harness that produced the recorded
legs (bench_users.py: closed loop, barrier, warmup, exact token-id prompts, ignore_eos), not
bench_async.py.

Conditions. Qwen3.8-27B UD-Q4_K_XL layer split over one ggml-rpc-server on the peer,
-ngl 99 -fa on -c 65536 --parallel 128 --cache-ram 0 -t 6 --kv-unified --device RPC0,CUDA0 -sm layer --tensor-split 0.5,0.5, 128 concurrent clients, 2 requests per client, npp 128,
ntg 256, both nodes pinned at 300,1690 and verified under load from clocks.sm sampled at
2 Hz (1678 to 1679 MHz local, 1683 MHz peer in every cell), not from clocks.max.sm, which reads
3003 either way.

leg	arm	tok/s	client tokens	server tokens_predicted_total
g2_b	two groups	190.05	65387	65544
g2_d	two groups	203.22	65390	65544
g2_e	two groups	193.62	65397	65544
g1_b	one context	116.14	65402	65544
g1_c	one context	115.01	65395	65544
g1_d	one context	116.71	65390	65544
g1_e	one context	117.48	65390	65544

Every cell served all 256 requests with n_err 0, and the server's own counter agrees with the
client to 0.23 percent (the residue is the harness's 8 token warmup plus the client figure coming
from agg_tok_s * wall_s).

Pooled: 195.63 against 116.34, ratio 1.682x at ntg 256, individual two-group legs at 1.634x,
1.665x and 1.747x. At ntg 128 the same setup reads 1.592x (167.59 against 105.25), the same
dilution direction this description already records.

Against the recorded legs: the one-context arm is within 1.9 percent (116.34 against 118.63)
and the two-group arm is 6.6 percent low (195.63 against 209.47). The gap is entirely in the
numerator.

What was eliminated, so that the remaining explanation is the credible one:

• not the buffer pool: LLAMA_STATE_BUFFER_POOL=1 is in the recorded harness's environment, but
  the string is absent from the recorded binary and from this one, so it was inert in both;
• not a CPU cap: the thermal guard capped the local CPU twice in the window, and none of the seven
  cells above overlaps a guard log entry (the two cells that did are discarded and named below);
• not a different layer boundary: resident memory was 20854 to 20926 MiB local and 20367 to
  20439 MiB peer in every cell, matching the recorded legs (20926 / 20439) to the megabyte;
• not the comment-trim commit: with gcc -fpreprocessed -dD -E and line markers dropped, all five
  files touched between a1dd7c5e8 and d98d90dd8 are token-identical, so this head is the same
  program the legs were taken on.

What is left is machine state. This block ran from 00:35 to 02:16 after eight hours of
continuous multi-agent load on the pair; the local node reached 82 to 86 C twice inside that
window. The mechanism this flag exploits is the host path, which is the part that suffers when the
package is hot and the memory system is busy.

Two cells were discarded and are in no mean: one where the thermal guard saw the peer at 90 C
and killed the peer's ggml-rpc-server mid-run (the master then aborted on the data-path assert
and half the requests came back truncated), and one where the guard capped the local node to
300,1200 mid-cell so the two nodes ran at 1365 and 1683 MHz. They would have read 143.58 and
106.31 tok/s. temperature.gpu was 64 to 66 C in both, so GPU temperature alone would have passed
them.

I am not editing the headline yet. A re-measurement on a genuinely rested pair, both arms inside
one hold, is queued; the number in the description will be set once, from that, whichever way it
comes out.