Skip to content

fix(routing): table pricing with catalog fallback for off-table models, pooled latency bootstrap, fresh tier cache - #12792

Open
maxmad64bis wants to merge 1 commit into
diegosouzapw:release/v3.8.51from
maxmad64bis:fix/batch7-pricing-db
Open

fix(routing): table pricing with catalog fallback for off-table models, pooled latency bootstrap, fresh tier cache#12792
maxmad64bis wants to merge 1 commit into
diegosouzapw:release/v3.8.51from
maxmad64bis:fix/batch7-pricing-db

Conversation

@maxmad64bis

Copy link
Copy Markdown
Contributor

⚠️ base-red inherited: #12732

Summary

Free models outside the price table were billed as premium, and tier labels couldn't see prices an operator had overridden in settings. This PR fixes both sides of that: routing now checks the free-model catalog (the list of providers with a documented free tier) before falling back to premium prices, and tier classification reads live database prices with a freshness check so it can't drift silently.

Say you run a free model that isn't one of the 21 hardcoded entries — before this change it inherited $5/$15 per million tokens and got pushed to the bottom of cost-aware routing. Now it resolves to $0 and routes as free. The change is additive: the sync hot path keeps serving table values, and the new async tier lookup falls back to it if the database is unreachable, so nothing gets slower or stricter.

It also stops guessing 1500ms for every unknown model's latency (the pool's observed median is used instead, with a counter tracking how often the guess path fires), scores real per-candidate costs instead of zero, and computes budget caps against actual estimated tokens rather than a fixed 1000.

Related Issues

Validation

  • Change type: routing
  • Focused tests and category gates from the golden path
  • npm run lint
  • Reconciled with the current active release base; focused checks rerun afterward
  • Production-code changes include a new or updated automated test in this PR

Tests Added Or Updated

  • tests/unit/pricing-fallback.test.ts (new, 4 tests): catalog free off-table → $0, prefixed id normalization, paid off-table → 5.0/15.0 fallback, table hits untouched
  • tests/unit/combo-bootstrap-latency.test.ts (new, 5 tests): median ignores corrupt entries, empty stats → undefined, counters, table-hit vs miss accounting, pure warn gate
  • tests/unit/tier-pricing-cache.test.ts (new, 4 tests): async tier reflects DB writes incl. resetAllPricing, DB-down fallback equals sync, sync path untouched by DB
  • tests/unit/virtualFactory-valuation.test.ts (new, 4 tests): reliability absent/null/0.2/errorRate, real table costs via helper
  • tests/unit/auto-combo-budget.test.ts (new, 2 tests): default math byte-identical, 32k tokens enforce 32x tighter budget via assert.throws
  • tests/unit/complexity-router.test.ts, tests/integration/manifest-routing.test.ts, open-sse/services/__tests__/manifestAdapter.test.ts: awaited for the new async tier lookups
  • scripts/check/check-pricing-freshness.mjs (new): STRICT gate failing when providerCostData.ts is untouched for 90+ days
  • npm run test:combo:matrix: 11 failures identical before/after this change (pre-existing on base, verified via stash comparison)
  • npm run lint: exit 2 on base and branch alike — pre-existing unused-var errors in combo.ts/manifestAdapter.ts/tierResolver.ts (none introduced here; full-suite count unchanged). Note: lint needed npm install es-abstract@1.23.9 --no-save --no-package-lock worktree-only to start at all (loader bug, noted per repo conventions).
  • No VPS round-trip needed: all changes are local pricing/scoring math covered by unit tests, no upstream behavior assumed.

Coverage Notes

  • Touches open-sse/ only: every new branch is covered by the tests above (table-first/catalog/fallback in getModelPricing, median + counters + warn gate, async tier + fallback counter, reliability/cost/budget in the virtual factory and engine).
  • No coverage gate regression attributable to this PR: the only untested lines are the fail-open catch paths, exercised implicitly by the DB-down test.

Reviewer Notes

  • Additive by design: classifyTier sync is untouched; getTargetTier stays sync; classifyTierAsync is migrated onto one real call-site (generateRoutingHints, awaited by its 3 consumers). If the async path ever misbehaves, the sync fallback returns byte-identical values.
  • Known behavior change (intended): virtual candidate rankings shift now that costs are real — cost-saver mode actually discriminates. Worth a glance at the resolveVirtualCost helper (open-sse/services/autoCombo/virtualFactory.ts).
  • The DEFAULT_MODEL_P95_MS table still has 8 entries (21 in the pricing table — separate tables, separate concerns); unlisted models now bootstrap from the pool median instead of a flat 1500ms, and the freshness gate keeps the pricing table honest going forward.
  • Snapshot reliability weight defaults to 0, so scoring only discriminates once mode packs carry it (fix(auto-combo): give every mode pack quality and reliability weights #12731).

@maxmad64bis
maxmad64bis force-pushed the fix/batch7-pricing-db branch 2 times, most recently from e41914b to 83b120f Compare September 5, 2026 10:52
…s, pooled latency bootstrap, fresh tier cache
@maxmad64bis
maxmad64bis force-pushed the fix/batch7-pricing-db branch from 83b120f to eda7c0c Compare September 5, 2026 11:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant