Skip to content

CONTEXT-262K.md

Latest commit

 

History

History
210 lines (150 loc) · 6.79 KB

CONTEXT-262K.md

File metadata and controls

210 lines (150 loc) · 6.79 KB
Raw

Doubling context to 262,144 — what changed and how to undo it

Date: 2026-07-29 Result: context 131,072 → 262,144 with no measurable speed loss. Cost: max concurrency at full context drops from ~9.6x to 4.81x.

Everything here was measured on the machine, not estimated.

Before / after

Before After
--max-model-len 131,072 262,144
Throughput (median, 5 warm runs) 74.8 tok/s 73.8 tok/s
Spread 74.4 – 75.3 73.6 – 74.4
Host memory used 80 GB 81 GB
GPU KV cache — 1,261,836 tokens
Max concurrency at full context ~9.6x 4.81x
--gpu-memory-utilization 0.60 0.60 (unchanged)

The 1.3% throughput difference is inside run-to-run noise. Generation speed on this model is bound by memory bandwidth against the 3B active parameters, not by how large the context window is allowed to be.

The real cost is concurrency, not speed

vLLM reported:

GPU KV cache size: 1,261,836 tokens
Maximum concurrency for 262,144 tokens per request: 4.81x

The KV cache is a fixed pool. Doubling the per-request ceiling halves how many full-length requests fit simultaneously. For a single user this is free — you never had 9 concurrent 131K conversations. If this box ever serves a team, that 4.81x is the number that binds, and the fix is more KV cache (raise --gpu-memory-utilization), not a smaller window.

Proof the context is real

Accepting a long prompt is not the same as being able to use it. Test: place a needle at the very start of a very long prompt and ask for it at the end.

prompt_tokens ACCEPTED: 208,848        ← 59% past the old 131,072 ceiling
prefill + decode wall:  55.1s
answer:                 'TANGERINE-4471'
needle recalled:        True

First attempt was wrong and is worth recording. The filler was sized at an assumed 4 chars/token, so it tokenized to only 119,374 — under the old limit. It proved nothing. This filler actually runs ~5.36 chars/token. If you re-run this test, verify usage.prompt_tokens exceeds the old cap before believing the result.

Tool-calling was re-verified after the change and still works:

{"name": "get_weather", "arguments": "{\"city\": \"Denver\"}"}

What was changed

1. Recipes (both, so either entry point is consistent)

~/spark-vllm-docker/recipes/qwen3-coder-next-lowmem.yaml
~/spark-vllm-docker/recipes/qwen3-coder-next-tuned.yaml

defaults.max_model_len: 131072 → 262144.

Backups: <recipe>.yaml.bak-131k alongside each.

2. systemd unit — a pre-existing drift, fixed

This was not part of the context change, but was found during it and would have caused a surprise on the next reboot.

  • vllm-coder.service was enabled but inactive — the running vLLM had been started by hand, not by systemd.
  • The unit pointed at qwen3-coder-next-**tuned** (gpu_memory_utilization: 0.80), while the process actually running was lowmem (0.60).
  • So a reboot would have started a different, untested configuration at higher memory pressure — and now at 262K as well, since both recipes were updated.

Changed to point at the recipe that was actually measured:

ExecStart=... run-recipe.py --solo qwen3-coder-next-lowmem

Backup: /etc/systemd/system/vllm-coder.service.bak-tuned

3. OpenCode client

~/.config/opencode/opencode.jsonc — "context": 131072 → 262144 for the spark-vllm provider. Backup: opencode.jsonc.bak-131k.

Restart OpenCode for it to take effect.

Rollback

Fast path — revert the running server only

ssh spark 'cd ~/spark-vllm-docker \
  && docker rm -f vllm_node \
  && nohup python3 run-recipe.py --solo qwen3-coder-next-lowmem \
       --max-model-len 131072 > /tmp/vllm-rollback.log 2>&1 &'

Wait ~2–4 minutes, then confirm:

curl -s http://10.0.4.93:8000/v1/models \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"][0]["max_model_len"])'

Full path — restore every file

ssh spark '
  cd ~/spark-vllm-docker/recipes
  cp qwen3-coder-next-lowmem.yaml.bak-131k qwen3-coder-next-lowmem.yaml
  cp qwen3-coder-next-tuned.yaml.bak-131k  qwen3-coder-next-tuned.yaml
  sudo cp /etc/systemd/system/vllm-coder.service.bak-tuned \
          /etc/systemd/system/vllm-coder.service
  sudo systemctl daemon-reload'

cp ~/.config/opencode/opencode.jsonc.bak-131k ~/.config/opencode/opencode.jsonc

Then restart vLLM using the fast path above (without --max-model-len, so the recipe default applies).

Restarting vLLM safely on this box

Three traps, all previously documented in TROUBLESHOOTING.md, all of which apply here:

  1. run-recipe.py silently skips cleanup if a container is already running, so you benchmark the old configuration while believing you changed it. The tell is a startup that reports READY in ~15 seconds — a genuine cold start on this model takes minutes. Always docker rm -f vllm_node first and confirm docker ps -a --filter name=vllm is empty.
  2. launch-cluster.sh stop can leave a stale container, causing a name conflict on the next launch. docker rm -f handles it.
  3. Never pkill -f by pattern — it matches your own SSH command and kills the session. Kill by PID.

Also: the first request after a cold start is slow (measured 17.4 tok/s vs 74 warm) because CUDA graphs are still being captured. Discard it. Benchmarking without a warm-up run will show a phantom regression.

Prerequisite that made this possible

Ollama was holding qwen3-coder:30b (25.3 GB) resident while unused. It released on idle timeout before the change, so no intervention was needed — but if memory is tight, force it:

curl -s http://10.0.4.93:11434/api/generate \
  -d '{"model":"qwen3-coder:30b","keep_alive":0}'

Check what is resident with curl -s http://10.0.4.93:11434/api/ps.

earlyoom was confirmed active before making the change. On this unified-memory box a GPU OOM can take the whole host down, so verify it first: systemctl is-active earlyoom.

Does more context solve the compaction problem?

Partly, and it is worth being honest about the limit.

Measured over ~17 hours of real OpenCode use against this server:

vllm:prompt_tokens_total       56,292,980
vllm:generation_tokens_total      244,361
                                  230 : 1

230 prompt tokens sent for every 1 generated. The window fills with conversation being re-sent every turn, not with model output. Doubling the window roughly doubles the turns before compaction — it does not escape the arithmetic.

The complementary fix is to make each fresh session cheap rather than to avoid ever starting one: a structural project map (~7K tokens for a 90-file project, bounded by the project's shape rather than its size) instead of a replayed transcript.