A cache-liveness scheduler for dual-agent (lead + sidekick) LLM harnesses.
The problem (per Devin Fusion): in a lead + sidekick harness, whenever one agent works the other idles, and provider prompt caches expire on idleness ("most cached inputs only have a 5-minute expiry"). A single expiry of the lead's cache can cost more than the delegation saved.
The idea: for each idle context the harness has exactly four actions, all with known, documented prices that let the cache die; keep it alive with pre-warm pings; pay for a longer TTL up front; or let it die and compact on resume. Picking the cheapest expected action per delegation is a small, fully-priced decision problem.
pip install --upgrade pip setuptools # once, for the editable install
pip install -e ".[dev]"
pytest # unit tests (no network, no API keys)
ruff check . # lint
cache-liveness-sim # the cost simulatorcache-liveness-sim replays synthetic delegation traces and compares five policies:
always let the cache die (naive), always ping, the scheduler (decides from a noisy
idle-time estimate), a fixed conservative policy, and an oracle that knows the real
idle time. It prints per-provider costs, fleet-level rate-limit load, and the
verification-plan accounting.
| Flag | Default | Meaning |
|---|---|---|
--runs |
500 | number of simulated sessions (traces) |
--delegations |
3 | lead→sidekick handoffs per session |
--sessions |
1000 | fleet concurrency for the rate-limit projection |
--noise |
0.5 | idle-time estimator error (log-noise sigma; 0 = perfect predictor) |
--seed |
7 | RNG seed — same seed, identical output |
Worth trying: --noise 0 (scheduler matches oracle), --noise 2 (scheduler loses its
edge to static policies), a different --seed (numbers shift, conclusions don't).
- On Anthropic the scheduler cuts lead-cache cost ~58% vs naive (oracle: 61%), and never does worse than the best static policy while estimator noise stays moderate (sigma ≤ 0.4; the edge disappears around 0.7).
- On OpenAI pre-5.6, free 24h retention nearly removes the problem (~83% savings); on GPT-5.6+ (30-min minimum TTL) most idles are covered anyway and every policy lands near 50%.
- At 1,000 concurrent sessions, always-pinging costs 0 rate-limited tokens/min on Anthropic (cache reads excluded from ITPM on most models) vs ~719M/min on OpenAI (cached tokens count toward TPM) the scheduler must be provider-aware, not just cost-aware.
- Idle time can also be productive: if the lead pre-writes the review checklist for the sidekick's in-flight work, review starts immediately on return. At default assumptions (1.5 turns saved when the plan holds, 30% staleness) this nets ~+900 tokens per delegation, flipping negative above ~37.5% staleness.
pip install -e ".[live]"
export ANTHROPIC_API_KEY="sk-ant-..."
python scripts/live_verify_anthropic.pyTakes ~16 minutes of wall time (deliberately like it proves TTL refresh and expiry in
real time) and costs about $0.02. It verifies four claims: an explicit
cache_control breakpoint writes the cache; an immediate follow-up reads it warm;
max_tokens: 0 pings keep it warm past the original 5-min TTL; and 5.5 minutes with
no pings lets it expire. A recorded passing run is in
scripts/live_verify_anthropic_results.json.
| Module | Contents |
|---|---|
cache_liveness/providers.py |
Verified pricing/TTL/rate-limit constants per provider, with doc citations |
cache_liveness/policy.py |
The pure decision engine + the fixed conservative v0 policy |
cache_liveness/heartbeat.py |
max_tokens: 0 pre-warm executor (Anthropic) + OpenAI analogue |
cache_liveness/simulate.py |
The trace-driven cost simulator |
tests/ |
Unit tests against hand-computed break-evens; simulator invariants |
All constants cited in providers.py:
Anthropic prompt caching ·
Anthropic rate limits ·
OpenAI prompt caching ·
OpenAI pricing