feat(warmup): proactive Claude warmup scheduler (#8848)

[HouMinXi]

Summary

Anthropic's 5-hour rolling window starts on the first inference request, so an account
that sits idle until 10am gives up an hour of capacity it could have had at 9am. #8848
proposed three ways to fix that; this implements Option B, the independent
warmupScheduler.ts the issue itself favours over folding warmup into token refresh.

A cron tick sends one max_tokens: 1 request to each opted-in Anthropic OAuth
connection, opening the window before real traffic arrives.

Off by default, and doubly gated. It needs OMNIROUTE_WARMUP_ENABLED truthy and
the connection listed in settings.claudeWarmup.connections. An empty connection list
warms nothing even with the env var on.

What a tick actually does:

• Cron evaluated in America/Los_Angeles (Anthropic's reset zone) regardless of the
  host clock. Default 0 7 * * *.
• Skips anything that is not a claude OAuth subscription connection — api_key auth,
  terminal statuses (banned/expired/credits_exhausted), connections in backoff, and
  connections already marked forbidden. Each skip logs its reason at debug level.
• POST /v1/messages with max_tokens: 1 and a rotating one-word prompt
  (hi/hello/ping/ready), so repeat ticks are not byte-identical payloads.
• 401 refreshes the token and retries once; 403 persists as forbidden and stops future
  attempts for that connection; 429 parses Retry-After.
• Circuit breaker with exponential backoff (5 min base, 240 min cap), Redis-backed with
  a SQLite fallback when Redis is unavailable. No hard Redis dependency is added.
• Undici bodies are drained on every path, error paths included.

Where this deviates from the issue

Three places, all worth your opinion rather than my assumption:

• The issue sketched interval_minutes: 300 plus an active_hours window. This uses a
  cron expression, which covers the issue's own "7am daily" example with less config
  surface. Happy to switch to the interval form if you prefer it.
• The timezone is fixed to America/Los_Angeles rather than configurable. That is
  Anthropic's reset zone, so a server elsewhere still fires at the right moment — but it
  is a deliberate narrowing of the issue's "server local time or configurable timezone".
• "Do not warm up if the account is already near its limit" is not implemented. The
  scheduler skips terminal, backed-off and forbidden connections, but never inspects
  remaining quota. One token against a near-limit account looked cheaper than a quota
  lookup on every tick, though that is a judgement call I would rather you made.

Related Issues

• Closes feat: proactive warmup scheduler for rate-limit window triggering #8848

Changes

• src/lib/warmupScheduler.ts — the orchestrator: tick, gating, classification, retry.
• src/lib/warmupScheduler/{core,backoff,circuitBreakerStore,redisCircuitBreakerStore,sqliteCircuitBreakerStore,circuitBreakerFactory}.ts
  — types, backoff curve, the two stores and the factory that falls back between them.
• src/lib/db/connectionRuntimeState.ts — column-level atomic UPSERT for warmup circuit
  state, plus src/lib/localDb.ts re-export.
• src/lib/db/migrations/135_connection_runtime_state.sql — new table only, no ALTER or
  backfill of anything existing. Slot 134 is 134_proxy_logs_egress_ip.sql on this base.
• src/shared/validation/settingsSchemas.ts, src/lib/db/settings.ts — claudeWarmup
  opt-in schema, defaulting to empty.
• src/lib/initCloudSync.ts — mounts the scheduler after startBudgetResetJob.
• .env.example + docs/reference/ENVIRONMENT.md — the four OMNIROUTE_WARMUP_* vars,
  added in the same commit as the code that reads them.

Tests Added Or Updated

• tests/unit/warmupScheduler.test.ts — orchestrator integration across each classified
  outcome, not just the success path: opt-in gating, Bearer + beta headers, message
  rotation, 401 refresh-and-retry, 403 forbidden persistence, 429 Retry-After,
  api_key skip, terminal-status skip.
• tests/unit/warmupScheduler/redisCircuitBreakerStore.test.ts
• tests/unit/warmupScheduler/sqliteCircuitBreakerStore.test.ts
• tests/unit/warmupScheduler/circuitBreakerFactory.test.ts — Redis-to-SQLite fallback.
• tests/unit/warmupScheduler/backoff.test.ts — curve and cap.
• tests/unit/db/connectionRuntimeState.test.ts — including the concurrent-write case
  the atomic UPSERT exists for.

Verification run locally:

$ npx tsx --test tests/unit/warmupScheduler.test.ts \
    tests/unit/warmupScheduler/*.test.ts \
    tests/unit/db/connectionRuntimeState.test.ts \
    tests/unit/check-env-doc-sync.test.ts
# tests 44
# pass 44
# fail 0

$ npm run check:migration-numbering  # OK
$ npm run check:db-rules             # OK
$ npm run check:cycles               # OK
$ npm run typecheck:core             # OK
$ npm run lint                       # OK

All three commits were also walked individually through
check:migration-numbering, check:db-rules and check:cycles, so the history
bisects clean rather than only being green at the tip.

Coverage Notes

Every new module under src/lib/warmupScheduler/ has a matching test file, and the
orchestrator is driven through a stubbed fetch across each outcome it classifies.

One deliberate gap: initCloudSync.ts gains a single startWarmupScheduler() call and
has no new test, since the existing bootstrap has no test harness.

Reviewer Notes

• connectionRuntimeState.ts uses a column-level UPSERT rather than
  read-modify-write. The tick runs up to 10 connections at a time, so two concurrent
  warmups would otherwise clobber each other's circuit state.
• The third commit is a fix worth reading on its own: the Redis store recorded success
  in Redis but left the SQLite backup's last_warmup_result at forbidden, so a Redis
  eviction would resurrect a stale block. clearWarmupCircuit does not touch that
  column, hence routing through upsertWarmupState instead. The test asserts the SQLite
  backup's final state, verified by injecting the regression and watching it fail first.
• Cost is one 1-token request per opted-in connection per tick — at the default cron,
  one per connection per day.
• check:file-size currently reports src/sse/handlers/chat.ts and
  open-sse/executors/base.ts over their frozen baselines. Neither file is in this
  diff, and both are byte-identical to release/v3.8.50, so those two are pre-existing
  on the base rather than anything this PR introduces.

[HouMinXi]

The failing checks here are inherited from the base, not from this diff. The run
on this PR went out at 15:00Z, before #9488 (14:36Z) and #9509 (15:53Z) cleared
the two base reds, and the same five checks (Fast Quality Gates, No new ESLint
warnings, Unit Tests 2/4, 3/4, 4/4) fail on #9482, #9483 and #9510, which share
no code with this one. I have no way to re-trigger the run from here.

Happy to rebase onto the current tip if you would rather see a clean run before
merging.

On priority: I have a follow-up branch ready that moves budget reset, token
health check and warmup onto a shared job registry, with run history and per-job
enable/disable. It builds on the scheduler shape introduced here, so this PR is
the one gating it.

[drewbitt]

UI for this?