You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d8cddbb
Browse filesBrowse the repository at this point in the historyBrowse files
Add prompt caching for agentic calls; stop pinning temperature
Agentic/multi-turn runs now inject Anthropic-style cache_control
breakpoints (via the new CachingOpenAIModel) on the system message and
the final message of every request, so providers that support prompt
caching reuse the growing conversation prefix instead of re-processing
it each turn. OpenAI-compatible proxies such as LiteLLM forward the
markers; providers that reject them get one uncached retry and the
(base_url, model) pair is remembered so later calls skip injection.
Controlled by a new prompt_caching config option (default on,
--prompt-caching/--no-prompt-caching on generate and config set).
Requests no longer send a temperature parameter anywhere (agentic
settings, call_llm, litellm/azure helpers, LLMBackend.complete):
some models only accept their default temperature and reject
explicit values, including 0.0.
Measured on a full self-documentation run through a LiteLLM proxy in
front of claude-sonnet: 73.8% of input tokens served as cache reads,
47% cost reduction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0 commit comments