You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5c0a7ab
Browse filesBrowse the repository at this point in the historyBrowse files
fix(xai): send body prompt_cache_key for Responses cache affinity (#2256)
* fix(xai): send body prompt_cache_key for Responses cache affinity
What: add a task-scoped `prompt_cache_key` to the xAI Responses API request
body in both `createMessage` and `completePrompt`, gated on the model's
`supportsPromptCache` flag and clamped to 64 characters.
Why: xAI's prompt cache is stored per server, so every turn of one conversation
must be routed to the same server to reuse it. For the Responses API the
documented affinity lever is the body field `prompt_cache_key`; the
`x-grok-conv-id` header is the Chat Completions / gRPC equivalent and is not
read on this path, so it was deliberately NOT added. Before this change
`xai.ts` sent no affinity identifier at all, leaving server-side cache routing
to load-balancer luck.
Impact: xAI turns of one task now pin to one cache server, raising cache-read
tokens and lowering cost on long conversations. The key is omitted (never sent
as an empty string) when no identifier is available or the model does not
support prompt caching. Usage accounting is unchanged: the shared normalizer
already reads `input_tokens_details.cached_tokens`.
* docs(api): attribute the 64-char prompt_cache_key cap honestly
What: reword the XAI_MAX_PROMPT_CACHE_KEY_LENGTH comment.
Why: review seat (PR #2256 mlarge P3) caught that xAI's docs do not state a 64-char bound for prompt_cache_key - the bound is OpenAI's documented one; we apply it as a conservative guard.
Impact: comment-only; zero code change.
0 commit comments