Describe the bug
With --cache-reuse 256 --slot-prompt-similarity 0.5 enabled on a multi-slot
unified-KV server, 4 concurrent chat-completion requests that share an identical
system prompt fail decode with:
E srv update_slots: decode() failed: structured KV live groups alias one F16 stage slot
E srv send_error: task id = N, error: decode() failed: structured KV live groups alias one F16 stage slot
All 4 tasks are then cancelled (stop: cancel task). Reproduced on two fresh
server restarts with the prose workload. A code-workload batch with identical
flags passed once after a failed batch, so it may be prompt-shape dependent.
Without --cache-reuse/--slot-prompt-similarity the same 4-stream load runs
cleanly (aggregate ~2.2x single-stream throughput).
Environment
- beellama.cpp v0.4.3 (
ba27edad2), native CUDA 13 / sm_120a build
- RTX 5090 32 GB
- Qwen3.8-27B IQ4_NL + mmproj-F16 vision
-ctk kvarn4 -ctv kvarn4 --kv-tail-tokens 1024 --kv-tail-type f16- spec
draft-mtp, n-max 3, draft K/V q4_0
-c 262144 --parallel 4 --kv-unified --no-cache-idle-slots -b 2048 -ub 1024 -ngl all -fa on
Reproduction
Server flags (relevant subset):
llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-F16.gguf \
-c 262144 --parallel 4 --kv-unified --no-cache-idle-slots \
-ctk kvarn4 -ctv kvarn4 --kv-tail-tokens 1024 --kv-tail-type f16 \
--spec-type draft-mtp --spec-draft-n-max 3 \
--spec-draft-type-k q4_0 --spec-draft-type-v q4_0 \
-ngl all -fa on -b 2048 -ub 1024 \
--cache-reuse 256 --slot-prompt-similarity 0.5
Fire 4 simultaneous /v1/chat/completions requests, each with the same
~200-token system message and distinct user messages (no shared user-prompt
prefix beyond the system message).
Expected: 4 concurrent generations complete.
Actual: all 4 fail with the error above; slots release with
n_tokens = ~1300-1450, truncated = 0.
Suspected cause
Interaction between KVarN structured KV live groups and prompt-reuse/similarity
slot selection: when multiple concurrently-processing slots share a common
prefix that gets reused via KV shifting/aliasing, live groups alias a single F16
tail-stage slot. Related context: the fork already carries a fix for unified
KVarN capacity sharing (856598ad1), which suggests this area is sensitive.
Happy to provide the harness script used to reproduce on request.
Describe the bug
With
--cache-reuse 256 --slot-prompt-similarity 0.5enabled on a multi-slotunified-KV server, 4 concurrent chat-completion requests that share an identical
system prompt fail decode with:
All 4 tasks are then cancelled (
stop: cancel task). Reproduced on two freshserver restarts with the prose workload. A code-workload batch with identical
flags passed once after a failed batch, so it may be prompt-shape dependent.
Without
--cache-reuse/--slot-prompt-similaritythe same 4-stream load runscleanly (aggregate ~2.2x single-stream throughput).
Environment
ba27edad2), native CUDA 13 / sm_120a build-ctk kvarn4 -ctv kvarn4 --kv-tail-tokens 1024 --kv-tail-type f16draft-mtp, n-max 3, draft K/V q4_0-c 262144 --parallel 4 --kv-unified --no-cache-idle-slots -b 2048 -ub 1024 -ngl all -fa onReproduction
Server flags (relevant subset):
Fire 4 simultaneous
/v1/chat/completionsrequests, each with the same~200-token
systemmessage and distinct user messages (no shared user-promptprefix beyond the system message).
Expected: 4 concurrent generations complete.
Actual: all 4 fail with the error above; slots release with
n_tokens = ~1300-1450, truncated = 0.Suspected cause
Interaction between KVarN structured KV live groups and prompt-reuse/similarity
slot selection: when multiple concurrently-processing slots share a common
prefix that gets reused via KV shifting/aliasing, live groups alias a single F16
tail-stage slot. Related context: the fork already carries a fix for unified
KVarN capacity sharing (
856598ad1), which suggests this area is sensitive.Happy to provide the harness script used to reproduce on request.