This repository was archived by the owner on Jun 8, 2026. It is now read-only.
FEA-1431→1436: adopt genai-prices as token-cost source of truth + cost reconciliation - #260
Merged
Merged
Conversation
Replace the hand-maintained pricing table + override pipeline (the source
of the v1 cached-token overcharge bugs) with @pydantic/genai-prices as the
single source of truth for model rates. This commit covers the desktop-main
cost path and the shared engine; the sidecar wiring + override-pipeline
removal in the build script follow in a subsequent commit.
- Add canonical token-cost engine in two parity-locked copies, because the
two cost paths run in different module systems that cannot share one file:
- scripts/agent-monitor-cost/cost-pricing.js (CJS, for the generated
sidecar tree) + its type:commonjs package.json scope.
- src/shared/token-cost.ts (ESM twin, for desktop-main, which must work
even when the sidecar process is disabled).
- Both wrap calcPrice/findProvider and return library prices UNCHANGED — no
rounding, clamping, asserting, or overriding. Their only job is feeding
correct INPUTS via the provider-aware input convention: Anthropic reports
input as fresh/uncached with additive cache fields (grand total = input +
cache_read + cache_write); OpenAI/others report input as the total with
cache as a subset (input passes through). This mirrors genai-prices' own
extractUsage and fixes the double-charging of OpenAI cached tokens.
- Repoint estimateTokenUsageCostUsd to the engine; it now returns
number | undefined and the loader omits the optional estimatedCostUsd
field when a model is unpriced (renders as "—", never a silent $0).
- Remove the now-dead PricingRow type, model_pricing query, sqliteLikeMatch,
and roundUsd helpers.
- Add test/token-cost.test.ts: asserts the twin and CJS engine return
byte-equal results across a fixture matrix, pins known dollar values
against the pinned library version, and guards CACHE_ADDITIVE_PROVIDERS
against the library's own extractUsage summing so drift fails loudly.
- Pin @pydantic/genai-prices to exact 0.0.62 and exempt it from the 7-day
supply-chain quarantine (audited; no install scripts, sole runtime
sub-dep is yargs).
Testing:
- pnpm exec tsx --test test/*.test.ts (1832 pass, 0 fail)
- tsc -p tsconfig.json (no errors)
- lint of changed source files clean (pre-existing test-file lint
unrelated to this change)
Risks:
- Cost values now come from genai-prices rather than the DB pricing table,
so historical estimates may shift to the library's (correct) rates.
- The sidecar still uses its old pricing.js until the follow-up commit
wires the engine into the generated tree; both land together in the PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Materialize the canonical token-cost engine (scripts/agent-monitor-cost/
cost-pricing.js, wrapping @pydantic/genai-prices) into the generated
server/lib so the pricing route resolves require("../lib/cost-pricing");
add it to the build stamp and hard-gate its presence.
- Rewrite patchPricingRoute: inject the engine require, replace upstream's
hand-maintained rule-table calculateCost with an engine-delegating version
(unpriced rows surface cost:null + a typed reason, never a silent $0; costs
returned at full library precision — no rounding, clamping, or override),
and remove the mutating PUT /api/pricing + DELETE /api/pricing/:pattern
endpoints (genai-prices is the source of truth — there is no host-editable
rule table to write to).
- Stop overriding upstream's DEFAULT_PRICING seed in patchDbFile and delete
the host pricing table (HOST_DEFAULT_PRICING) + renderDefaultPricingSource.
The hand-maintained table was the root cause of the cached-token overcharge
bug. Upstream's own DEFAULT_PRICING stays intact (still exported via the
module.exports anchor, still seeds the read-only model_pricing listing) but
no longer feeds any cost calculation.
- Add assertGeneratedTree hard-gates: engine present in server/lib, pricing
route requires the engine and uses the delegating calculateCost, the old
per-mtok formula is gone, and PUT/DELETE are absent — so a future upstream
bump can't silently revert to the overcharge-prone path.
Testing:
- pnpm build:agent-monitor passes (SQLite gate + all hard-gates green);
generated pricing.js verified (engine require, delegating calculateCost,
no PUT/DELETE).
- Engine resolves @pydantic/genai-prices from the generated server/lib and
prices the additive-cache anthropic fixture to $0.007125.
- pnpm typecheck clean; eslint src/ clean; pnpm test green (1832 desktop unit
+ 104 pack/PR module tests).
Risks:
- Removing PUT/DELETE /api/pricing is an internal sidecar-only contract change
(consumed only by the bundled client, shipped as one unit) — no external
migration required. The bundled client's pricing-edit UI is replaced by the
read-only catalog in FEA-1433 (same PR).
- Costs are now full-precision floats (previously rounded to 4 dp); display
formatting is handled in the UI layer (FEA-1433).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ecar
Server-side foundation for "model not priced" handling. Builds on the
FEA-1431 engine-delegating calculateCost (which already returns cost:null
for unpriced rows instead of a silent $0).
- patchSessionsRoute: alongside row.cost, populate row.unpriced_models from
the engine breakdown (deduped names of rows the engine could not price), at
both list code paths (price-sort and default order). Additive field on an
internal sidecar response consumed only by the bundled client — no external
contract migration required.
- patchPricingRoute: GET /api/pricing now stamps the pinned genai-prices
version (engine: { name, version }) so the read-only catalog can label its
single source of truth. Version is injected at build time by reading the
installed package.json directly — the package's exports map blocks
require("@pydantic/genai-prices/package.json") at runtime. resolveGenaiPricesVersion
fails the build loudly if the version is missing (no silent null stamp).
- assertGeneratedTree: hard-gate both patches so a future upstream bump that
breaks an anchor fails the build instead of silently dropping the signal.
Testing:
- pnpm build:agent-monitor (all hard-gates pass; SQLite gate PASS)
- node --check on generated sessions.js + pricing.js (valid)
- verified generated output: row.unpriced_models at both call sites,
const GENAI_PRICES_VERSION = "0.0.62", engine stamp on GET /api/pricing
- pnpm typecheck, pnpm lint, pnpm test (1832 tsx + 104 pack/PR, all pass;
token-cost parity 5/5)
Risks:
- unpriced_models is additive; older bundled clients ignore it. The desktop
client ships in the same Electron build, so producer/consumer update atomically.
- Version stamp is a build-time literal; it tracks the installed package, not a
range — intentional, so the UI reflects exactly what is bundled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The server now populates session.unpriced_models (models the canonical token-cost engine, genai-prices, cannot price). Render it in the Sessions table cost cell so an unpriced model never collapses to a silent "$0"/"-": - Fully priced -> the dollar total (unchanged). - Partially priced (some models unpriced) -> total + amber "partial" badge. - Fully unpriced -> amber "not priced" badge (no fabricated $0). - The badge tooltip names the exact unpriced models. - Legitimately-zero / no-token sessions still render "-". Testing: - pnpm build:agent-monitor (Vite client build + SQLite gate PASS) - pnpm typecheck (no errors), pnpm lint (clean) - pnpm test (tsx suite + 104 pack/PR tests, 0 fail) Risks: - Pure presentation change in the bundled sidecar client overlay; no server, schema, or HTTP-contract change. unpriced_models is optional so older session payloads simply render as before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… catalog
The hand-editable pricing table was removed in the genai-prices migration,
so the Settings pricing editor's add/edit/delete actions (PUT/DELETE
/api/pricing) now 404. This replaces that editor with a read-only catalog
that reflects genai-prices as the single source of truth for rates.
- Overlay scripts/agent-monitor-client/Settings.tsx (full-file override,
registered in build-agent-monitor.mjs CLIENT_FULL_FILE_OVERRIDES).
- Remove the pricing CRUD UI (add/edit/delete rows, reset-to-defaults) and
its state/handlers; drop the now-unused ModelPricing type + icon imports.
- Render the pinned engine version stamp ("@pydantic/genai-prices vX.Y.Z")
from GET /api/pricing's engine field.
- Show a per-model priced/unpriced breakdown from the cost endpoint: priced
rows display input/output/total cost; unpriced rows surface an amber "not
priced" badge with the engine's unpriced_reason (never a silent $0) and a
footnote that unpriced models are excluded from totals.
Testing:
- rtk pnpm build:agent-monitor (Vite bundle + SQLite gate PASS)
- rtk pnpm typecheck (no errors)
- rtk pnpm lint (clean)
- pnpm test (104 pass, 0 fail)
- Verified no stale references to removed symbols remain in the overlay.
Risks:
- Client overlay is bundled by Vite and is not type-checked/linted by the
desktop gates; mitigated by grep-verifying removed imports are single-use
and that build/runtime references resolve. The overlay cpSync throws if the
source file is missing, protecting against a silently-dropped override.
- Internal contract only (sidecar's own bundled client) — no migration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foundation slice for per-session billing-mode detection and the two-ledger
(metered API vs subscription-covered) accounting. Mirrors the proven
agent-monitor-cost engine+twin+parity pattern so the same classification runs
in both the CommonJS sidecar tree and ESM desktop-main (which must work with
the sidecar disabled).
- scripts/agent-monitor-billing/billing-mode.js: canonical CJS engine.
Classification (BILLING_MODES, billingLedger, isMeteredApi, isSubscription,
normalizeBillingMode) is total over the union; detection
(detectBillingModeForHarness + per-harness helpers) is dependency-injected
({ env, fileExists, homeDir }) and existence-only.
- scripts/agent-monitor-billing/package.json: scopes the dir to type:commonjs
within the type:module parent (mirrors agent-monitor-cost).
- src/shared/billing-mode.ts: byte-equal ESM twin with typed BillingMode /
BillingLedger / BillingModeDetectionDeps.
- src/main/billing-mode-detector.ts: wires real process.env + existsSync +
os.homedir() for desktop-main (sync-time fallback for legacy/unknown rows).
- test/billing-mode.test.ts: parity (twin vs engine), the exact reviewed
ledger invariant (subscription modes never classify as metered), the
existence-only detection matrix incl. CODEX_HOME override, and a guard that
detection never surfaces a secret value.
Secret-handling: detection checks credential EXISTENCE only — never reads the
contents of ~/.claude/.credentials.json, ~/.codex/auth.json, or any API-key
env var beyond a non-empty check, and never logs/returns those values.
Testing:
- pnpm typecheck: no errors
- pnpm lint: clean
- pnpm test: 104 prior + 6 new billing-mode tests pass, 0 fail
Risks:
- Pure foundation: not yet wired into schema, sync contract, importers, or UI
(subsequent FEA-1434 slices). No behavior change ships from this commit
alone. Tier-specific Anthropic values (pro/max_5x/max_20x) exist in the
union for the persisted/synced contract but are not yet emitted by
existence-only detection (await /status parsing, out of scope per PRD-414).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Materialize the billing-mode engine into the generated agent-monitor server/lib (BILLING_MODULES), hash it for cache-busting, and hard-gate its presence in assertGeneratedTree. - Patch the generated db.js with an idempotent billing_mode column migration (try-SELECT / catch-ALTER, default 'unknown') + index, plus a setSessionBillingMode prepared statement, anchored on the existing harness block; throws if anchors are missing. - Add optional billingMode to the SyncedAgentSession cloud-relay contract. Additive only: schema version is unchanged, older builds omit the field, relay treats absent as "unknown" (not a breaking change). - Surface billingMode from loadSyncedSessions: use the stored value when known, otherwise fall back to live detection by harness. Testing: - pnpm build:agent-monitor (SQLite gate PASS); generated db.js verified to carry both harness + billing_mode migrations and both prepared statements. - pnpm typecheck (no errors); pnpm lint (clean). - pnpm test green, including new wiring-static + sync-service cases (explicit codex_subscription propagation; legacy-unknown live fallback). Risks: - Generated-tree patch relies on string anchors; build hard-gates fail loud if upstream drifts. Contract change is additive so no relay migration is required. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add a shared write-path helper (agent-monitor-shared/billing-stamp.js) that detects the harness billing mode via the canonical engine and persists it through setSessionBillingMode. One entry point, used by every harness, so detection isn't duplicated per importer. Detection is existence-only (no credential contents read) and best-effort (failures are swallowed so a stamp never blocks a hook or import). - Materialize the helper into the generated server/agent-monitor-shared via SHARED_MODULES (hashed for cache-busting, existence hard-gated). - Patch the Claude hook route (patchHooksBillingMode): stamp 'claude' right after the session_created broadcast in ensureSession, so the mode is set before any token-usage events flow. Hard-gated in assertGeneratedTree. - Stamp the four non-Claude importers (codex/cursor/copilot/opencode) immediately after setSessionHarness. Testing: - pnpm build:agent-monitor (SQLite gate PASS); verified the generated hooks.js stamp + materialized helper + all four importer stamps. - pnpm typecheck (no errors); pnpm lint (clean). - pnpm test green (1841 tsx + 104 node), incl. a new wiring-static case asserting every harness stamps via the shared helper and the soak catchup-cache test which exercises the generated importer -> billing-stamp -> billing-mode require chain at runtime. Risks: - Adds one existence-only credential check per imported session inside the import transaction; OS-cached and trivial, mirroring the existing per-session setSessionHarness call. Hook-route + helper wiring is build-time hard-gated, so an upstream bump that breaks an anchor fails the build rather than silently leaving sessions unstamped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Server accounting core for the two-ledger model. The headline cost is now
the real-spend total only — subscription-covered spend (a hypothetical
"would have cost") is surfaced separately and never inflates it.
- Add pure ledger-accounting helpers to the billing-mode engine + its
byte-equal desktop-main twin: emptyLedgerTotals(), addLedgerCost()
(routes a priced row into its ledger via billingLedger; ignores
non-finite costs so an unpriced row can't corrupt a total), and
headlineCost() (metered + unknown, EXCLUDING subscription).
- Patch the analytics headline (GET /api/analytics) and the all-sessions
GET /api/pricing/cost to LEFT JOIN sessions, bucket each row's cost by
billing_mode, key total_cost off headlineCost, and expose cost_by_ledger
{metered, subscription, unknown}. LEFT JOIN keeps orphan usage rows in the
"unknown" ledger rather than silently dropping them.
- Per-model breakdown + daily_costs on /cost are unchanged (additive split).
- Per-session GET /cost/:sessionId is intentionally untouched — its detail
view needs the full per-session cost (the subscription "would have cost"),
which Slice 4b labels via the session's billing_mode.
- Hard-gate both generated routes (require resolved, cost_by_ledger present,
headline keyed off headlineCost, LEFT JOIN applied, upstream un-joined scan
gone) so a future upstream bump can't silently ship an un-split headline.
Testing:
- pnpm typecheck (clean), pnpm lint (clean)
- pnpm test: 1847 + 104 pass (+6 new ledger parity/invariant tests)
- billing-mode.test.ts pins headline = metered + unknown across the full
mode domain and proves subscription cost never moves the headline
- agent-monitor-wiring-static.test.ts gates both patched routes in the
generated tree
- Rebuilt the sidecar (build:agent-monitor) — all hard-gates + SQLite gate
PASS; verified generated billing-mode exports the helpers and the route
usage path computes headline correctly
Risks:
- Headline real-cost will DROP for users with subscription sessions — this
is the intended behavior change, not a regression.
- breakdown sum on /cost no longer reconciles with total_cost when
subscription models are present (breakdown is full per-model detail;
total_cost is the headline). Slice 4b renders the ledger split in the UI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Slice 4b — the client UI for the metered-vs-subscription split landed server-side in 0790889. All cost math stays server-owned; the client only reads the already-computed cost_by_ledger and labels subscription sessions. - New shared overlay scripts/agent-monitor-client/lib/closedloop-ledger.ts: the localStorage "show hypothetical API cost" pref (Settings writes, Dashboard reads), the CostByLedger response shape, and a presentation-only mirror of the server SUBSCRIPTION_MODES set for badge labelling. No dollars are computed here — drift can only mislabel a cosmetic badge. - StatusBadge gains BillingBadge, rendered only for subscription sessions (no fabricated quota %); Sessions renders it from session.billing_mode. - Dashboard Total Cost pill value stays the billed headline (metered + unknown); the subscription "would have cost" appears only in the pill subtitle, gated by the opt-in toggle, never summed into the headline. - Settings adds the opt-in toggle (default off), persisted via the shared helper. - build-agent-monitor.mjs: deliver the ledger helper via CLIENT_FULL_FILE_OVERRIDES, add an additive billing_mode?: string|null edit to the Session type, and fix a latent cache bug — currentStamp() omitted the Dashboard/Settings overlays, so editing only those left the cached generated tree stale. It now hashes every full-file client overlay. Testing: - rtk pnpm build:agent-monitor --force (Vite resolves the overlay imports; the billing_mode type edit anchor matched; SQLite gate PASS; bundle contains the BillingBadge labels + "subscription-covered" subtitle). - rtk pnpm typecheck (no errors), rtk pnpm lint (clean). - pnpm test — 104 pass, incl. a new closedloop-ledger.test.ts that asserts the client subscription mirror agrees with the canonical billing-mode engine across the full BILLING_MODES domain (anti-drift), plus a Slice 4b wiring-static test for the overlay/stamp/type wiring. Risks: - Client overlays are not typechecked (esbuild strips types); mitigated by the build-must-run-after-edit gate and the wiring-static needles. - The badge mirror could drift from the server set; the cross-module domain test fails CI if they diverge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ation Pure, dependency-free foundation for nightly cost reconciliation (FEA-1435/1436), mirroring how FEA-1434 began with the billing-mode engine. No I/O, no DB, no network, no pricing — both modules are ESM src/main so they are typechecked. - cost-math.ts: convert vendor/local amounts ONCE to integer micro-cents at the boundary (1 cent = 10_000, 1 USD = 1_000_000), then aggregate/diff with exact integer arithmetic. usdToMicroCents (sidecar estimate, OpenAI USD), parseDecimalCentsToMicroCents (Anthropic decimal-string cents, parsed without a whole-magnitude float multiply), centsToMicroCents, microCentsToUsd, sumMicroCents, computeDrift (signed micro-cents + percentage, null when vendor billed $0). Every result is Number.isSafeInteger-guarded so an implausibly large input fails loudly rather than silently losing precision. Plain number (not BigInt) since realistic spend is far below 2^53; documented. - reconciliation-cause-hint.ts: pure rankDriftCauses(features) returning ranked, human-readable causes for the Diagnostics explain expander. Drift direction selects candidates; the Anthropic 1h cache-write gap is flagged permanent and links to the upstream genai-prices project homepage (no fabricated issue number); "unknown" is always retained as a fallback. Neither module prices anything — pricing stays the sole responsibility of token-cost.ts / the sidecar engine, which TRUST genai-prices for every rate. Testing: - pnpm typecheck: clean - pnpm lint: clean - pnpm test: 104 pass / 0 fail (16 new across cost-math + cause-hint), covering conversion factors, decimal-string precision + half-up carry, exact integer summation, safe-integer guards, signed drift / null-percentage, and direction- based cause ranking incl. the permanent cache-write gap link. Risks: - Foundation only; not yet wired to any worker, DB table, or UI (later slices). - Micro-cent rounding loses sub-micro-cent precision by design (finer than any vendor reports). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Persists one row per (day x vendor x model): the local genai-prices estimate, the vendor-billed amount, and the drift between them (integer micro-cents). All of FEA-1435/1436 lives in desktop-main + host renderer + IPC — the sidecar is deliberately NOT involved (see below), so no generated-tree patch, sidecar route, or DB twin is needed. - reconciliation-store.ts: ReconciliationStore wraps electron-store (same convention as activity-log-store). upsert keyed on (day,vendor,model) so a re-run replaces rather than duplicates; list(query) filters by day range + vendor, newest first; retention prunes rows older than the window (injectable clock) with a hard size cap; corrupt/malformed persisted rows are runtime- validated and dropped on load (money fields guarded as safe integers) so a bad file degrades to "no data" rather than poisoning diagnostics. - Dumb storage only: NO pricing, NO drift math (the worker computes rows via token-cost.ts + cost-math.ts and hands them in fully formed). Why main-only / no sidecar: reconciliation depends on vendor Admin API keys (OS keychain via safeStorage, main-process only) and outbound billing-API calls. The sidecar is a sandboxed localhost process whose client runs in an iframe; org billing data and anything derived from Admin keys must never transit it. Main reads token usage from dashboard.db READ-ONLY (as the sync service already does), persists results here, and will serve them to the host renderer over IPC. Testing: - pnpm typecheck: clean - pnpm lint: clean - pnpm test: tsx suite 1874 pass / 0 fail (6 new), covering upsert-replace, range/vendor query order, retention pruning, corrupt-row rejection on load, file round-trip, and clear. Risks: - Storage only; not yet wired to a worker or IPC (later slices). - Retention default 90 days / 5000-row cap; informational data, safe to prune. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ation - Add AdminKeyStore: a single safeStorage-backed class parameterized by vendor, holding the org-level Admin API keys (Anthropic sk-ant-admin…, OpenAI sk-admin…) that nightly reconciliation uses to read vendor billing. - Keep keys main-process only, encrypted at rest in a dedicated electron-store file (desktop-admin-secrets), namespaced per vendor so the two never collide and clearing one leaves the other intact. - Expose a two-method contract mirroring api-key-store.ts: getKey() returns plaintext for main-process outbound vendor calls ONLY (never IPC/logged); getStatus() is existence-only (hasKey: boolean) and is the only shape that may cross IPC to the renderer. - Add thin createAnthropicAdminKeyStore / createOpenAiAdminKeyStore factories so the safeStorage plumbing is not duplicated per vendor. Testing: - New test/admin-key-store.test.ts (8 cases): round-trip + existence-only status, plaintext never written to disk, rehydration, per-vendor namespacing isolation, clearKey, empty/whitespace rejection, trimming, and safeStorage-unavailable degradation. - pnpm typecheck clean, eslint clean, full suite green (1882 + 104, 0 fail). Risks: - Low. New isolated module with no callers yet; admin clients (Slice D) and IPC (Slice F) will consume it. No external/persisted contract is exposed to older app versions — the store file is internal and main-only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion - Add admin-billing.ts shared foundation: the normalized VendorBilledEntry shape, an injectable AdminFetchLike surface, the assertAllowedAdminHost network allowlist (https + exact vendor host only), a redacting requestAdminJson helper, and the day-from-timestamp + money-unit converters. - Add AnthropicAdminClient: GET /v1/organizations/cost_report with group_by[]=description and bucket_width=1d; parses amount as DECIMAL-STRING CENTS (verified) via parseDecimalCentsToMicroCents; per day×model with server-side tool costs reported faithfully (model null). - Add OpenAiAdminClient: GET /v1/organization/costs with bucket_width=1d and NO group_by — the endpoint has no per-model dimension, so we return the day TOTAL (model null) and let the worker reconcile OpenAI at day grain rather than fabricate a split the vendor never provided. amount.value is a USD number (verified) → usdToMicroCents. Security/correctness: - Admin key travels ONLY in request headers (x-api-key / Bearer), never the URL/query; errors include HTTP status + the vendor's (key-free) error body, never the key. - Every request host is checked against the allowlist before fetch. - Pagination follows next_page; exceeding a hard page cap THROWS rather than returning a partial (understated) bill that would manufacture false drift. - Malformed/missing money throws rather than silently dropping a charge. Testing: - New test/admin-cost-clients.test.ts (10 cases) with a recording fake fetch (no network): host-allowlist accept/reject (scheme, look-alike host, junk), decimal-cents and USD-number → micro-cents conversion, day/model tagging, key-in-header-not-URL, next_page pagination, page-cap abort, HTTP-error and malformed-money rejection, empty-key construction guard. - pnpm typecheck clean, eslint clean, full suite green (1892 + 104, 0 fail). Risks: - Live wire formats were verified against the public API docs (Anthropic decimal-string cents + group_by[] bracket form; OpenAI USD-number, Unix seconds, no per-model grouping). Mocked tests cannot catch a future vendor-side schema change; the defensive parser fails loud if one occurs. - No callers yet; the reconciliation worker (next slice) wires store → client. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add reconciliation-worker.ts: aggregates local metered API spend from token_usage+sessions (effective raw+baseline tokens, priced via the genai-prices engine in token-cost.ts) and reconciles it against vendor cost-report bodies, emitting drift notices with ranked cause hints. - Reconcile per vendor grain: Anthropic per-model (tool-cost rows with model=null fold into a "(server-side tools)" sentinel that explains under-estimates as server_side_tool_use); OpenAI collapses all models to a day-grain sentinel since its cost API has no per-model dimension. - A vendor with no fetch fn (no Admin key configured) is NOT reconciled against a $0 bill, so a missing key never manufactures a false "vendor billed nothing" drift notice. - loadMeteredUsageRows reads the live dashboard.db (JOIN token_usage/sessions), sums raw+baseline per token field, resolves billing_mode, and filters to metered API rows only — non-metered subscription usage is never priced. - DRY: extract shared electron-free resolveBillingMode into billing-mode-detector.ts; agent-session-sync-service.ts delegates to it so the worker reuses the exact billing-mode rule without pulling electron into its test path. Testing: rtk pnpm typecheck (clean); rtk pnpm lint (clean); pnpm test (both runners green: 1904 + 104 pass, 0 fail); reconciliation-worker.test.ts 12/12 pass (per-vendor grain, drift thresholds, cause-hint mapping, metered-only filtering, real ReconciliationStore persistence, raw+baseline token sums over a live SQLite DB). Risks: worker is pure/injectable and not yet wired into production (no IPC, no Admin-key plumbing) — that lands in the next slice. No outbound vendor calls happen until a fetch fn is supplied. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add loadMeteredUsageRowsFromDisk to the worker: opens dashboard.db READ-ONLY with the sync service's DatabaseSync pattern (busy_timeout, close in finally), loads metered rows since the cutoff, returns [] when the DB file does not exist yet so a fresh install reconciles nothing instead of throwing. - Add CostReconciliationService: the seam between the pure worker and the live app. Owns the two vendor Admin key stores and the reconciliation store, builds the vendor cost clients from stored Admin keys, runs the worker, and schedules it (initial delayed run + nightly interval). - Reconcile each configured vendor in its OWN worker pass (only that vendor's fetch fn wired in) so a bad key / 401 / network error is isolated per vendor and never aborts the other or degrades to a fabricated $0 bill. - Admin keys are read via getKey() ONLY to construct outbound clients; the only key shape that leaves the service is the existence-only AdminKeyStatus. - Runs are serialized (skippedBusy) so a manual run cannot race the scheduled one; usage is loaded once per run and reused across vendor passes. - Service imports the stores as TYPES only (concrete classes pull in electron-store/safeStorage), keeping it unit-testable under tsx with no Electron. Testing: rtk pnpm typecheck (clean); rtk pnpm lint (clean); cost-reconciliation-service.test.ts 11/11 (status existence-only, set/clear round-trip, per-vendor pass + summing, per-vendor error isolation, no-key vendor never queried, no-keys no-op, single usage load, listRows passthrough, skippedBusy race, start/stop scheduling via injected timer seams); reconciliation-worker.test.ts still 12/12 after the value-import change. Risks: not yet reachable from the UI (no IPC/preload/renderer) — that lands in the next slice. No outbound calls until an Admin key is configured. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Instantiate CostReconciliationService in DesktopApplication with the real vendor Admin key stores, the reconciliation store, and a loadUsageRows that re-opens dashboard.db READ-ONLY each run (windowed via reconciliationCutoffIso). - Schedule it on boot (independent of the Agent Monitor toggle — the tick no-ops without a key and loadUsageRows returns [] when there is no DB) and stop it in the shutdown sequence alongside the other background services. - Add IPC handlers (get-admin-key-statuses, set-admin-key, clear-admin-key, run-cost-reconciliation, list-cost-reconciliation) delegating to the service, plus the matching preload bridge methods. - Only existence-only statuses, persisted drift rows, and key-free run summaries cross IPC — the Admin key material never does (read main-side only to build the outbound vendor clients). - Runtime-validate the vendor and list-query IPC inputs before use (untrusted boundary): an unknown vendor throws; a malformed query degrades to "all rows". Testing: rtk pnpm typecheck (clean); rtk pnpm lint (clean); pnpm test (both runners green: 1915 + 104 pass, 0 fail); service test stable across 3 runs. Risks: no renderer surface yet — the handlers are reachable over IPC but there is no Settings/Diagnostics UI to drive them; that lands in the next slice. The nightly scheduler is live, but it stays idle until an Admin key is configured. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…t table) - New "Cost" Settings sub-tab in the host renderer with vendor Admin key intake for Anthropic (sk-ant-admin…) and OpenAI (sk-admin…): Save & Verify, Clear, and an existence-only status line per vendor. - "Save & Verify" sets the key then runs a reconciliation pass and surfaces that vendor's auth/network error (401/403/etc.) inline from the run summary's errors[] — a bad key reports "verification failed" without ever echoing the key. - Drift Diagnostics table (day | vendor | model | local | vendor | drift $/%) from listCostReconciliation, with a 30-day aggregate drift readout, a manual "Run reconciliation now", a Refresh, and an "Export for support" JSON download built from the in-memory rows + notices. - Per-row "Explain" expander shows the run's ranked cause hints (incl. the permanent "expected" 1h-cache-write gap). Hints come only from a fresh run summary — persisted rows without a hint show drift numbers but no fabricated cause. The cause link renders as plain text since the renderer has no external-URL open handler. - All cell values rendered via textContent (no HTML injection from model ids); the bridge only ever moves existence-only statuses, persisted rows, and key-free summaries. - Bump desktop version (main caught up to 0.15.99). Testing: rtk pnpm typecheck (clean); rtk pnpm lint (clean); pnpm test (both runners green: 1915 + 104 pass, 0 fail). Renderer is static HTML/JS, not covered by tsc/eslint/tests — verified by manual read-through against existing Settings sub-tab patterns. Risks: renderer JS has no automated coverage; the panel wiring mirrors the existing api-key intake + feature-flags patterns exactly. Cause hints are not persisted, so they only appear after a run in the current session (by design — we never invent a cause for a persisted row). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add read-only ClaudeCodeAnalyticsClient for GET /v1/organizations/ usage_report/claude_code: single-day-per-request loop over the window, per-day cursor pagination, x-api-key header (key never in URL), strict money parsing (estimated_cost.amount cents -> micro-cents), tolerant actor mapping (user email / api key name / unknown), per-day page cap. - Add ClaudeCodeAnalyticsService seam: shares the Anthropic Admin key store with reconciliation via a minimal getKey/getStatus reader (ISP), computes the trailing window from an injectable clock, clamps renderer windowDays, and surfaces fetch failures as key-free errors instead of throwing. - Wire IPC (desktop:get-claude-code-analytics) + preload bridge with an untrusted-payload validator; add a "Claude Code Usage" group to the Cost settings tab showing per-user spend aggregated from the records. - Extract the fake-fetch test helper into test/helpers/admin-fetch.ts so the admin-cost and analytics client tests share one recorder. This estimate is Anthropic's own and is shown for reference only; it never overrides the local genai-prices ledger or feeds the reconciliation math. Testing: - pnpm typecheck: clean; pnpm lint: clean. - pnpm test: 2033/2033 pass (15 new: 10 analytics client, 5 service). - New client tests cover header-only key, per-day looping, pagination, non-2xx, malformed money, page-cap, unknown actor, window validation. Risks: - Renderer (single-file HTML/JS) is not covered by tsc/eslint/tests; validated by manual read-through against the existing Cost-tab patterns. - Endpoint requires a Team/Enterprise org; non-eligible orgs surface as an empty table with an explanatory message rather than an error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Independent pre-PR review (two reviewers) returned SHIP WITH MINOR FIXES. This commit resolves the actionable findings: - admin-billing: redact key-shaped tokens (sk-…) from vendor error bodies before they reach a thrown error, IPC reply, or the log file. OpenAI's 401 body echoes a copy of the key it received; the vendor response is outside our control, so requestAdminJson now scrubs it via redactKeyLikeTokens. Fixes the "key never in error message" invariant and corrects the now-accurate comments. - claude-code-analytics-client: tokenCount rejects negative values (matches its non-negative contract); assertUtcDayString round-trips the parsed date so an impossible calendar day (e.g. 2026-02-30, which JS silently overflows) is rejected instead of being silently shifted. - renderer: stop shadowing the browser `window` global in describeClaudeCodeResult; derive the usage-total label from the actual query window instead of hardcoding "7-day total". - reconciliation-worker: document why the dashboard.db handle is intentionally opened read-write (a SQLITE_OPEN_READONLY connection cannot attach to the sidecar's live WAL database without an existing writer) — declined the readOnly:true suggestion to avoid an intermittent open failure. Testing: - pnpm typecheck: clean - pnpm lint: clean - pnpm test: 2035 pass / 0 fail (added redaction, negative-token, impossible-date, and service key-absence regression tests asserting the exact reviewed invariants) Risks: - redactKeyLikeTokens over-redacts by design (any sk-… run → sk-[redacted]); diagnostic bodies stay readable, secrets cannot leak. - reconciliation read path unchanged (read-write open, SELECT-only) — no behavioral change to WAL coexistence. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
closedloop-electron/apps/desktop/src/main/reconciliation-worker.ts
Lines 305 to 306 in d446431
Base reconciliation windows on raw usage days
When every metered row in the period is for a model that computeTokenCost cannot price, aggregateLocal() drops all of those rows, so localDays is empty and neither vendor Admin API is queried. In that scenario reconciliation silently records no vendor-billed under-estimate even though there is local usage and an Admin key; this is especially likely right after a provider releases a model before genai-prices knows it. Derive the fetch window from the metered usageRows dates, not only from priced local cells, so vendor-only cells can still surface the missing local estimate.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
thadeusb
approved these changes
May 29, 2026
thadeusb
left a comment
thadeusb
left a comment
Contributor
There was a problem hiding this comment.
Money math, admin-key handling, and the twin-engine parity are all tight, and the build hard-gates mean a future upstream bump can't silently drop the billing dimension. Only real gap is the windowing one Codex already flagged. Settle that and ship.
…agination, honest key verification Codex PR review surfaced one [P1] and two [P2] findings; all three were verified against the real code (the [P1] reproduced with a live Node fetch test proving the Admin key leaked into the thrown error) and fixed here. - [P1] Admin-key header-safety: admin-key-store.setKey now rejects any key with non-header-safe bytes (outside 0x21–0x7E) at the untrusted IPC boundary, so a control char / non-ASCII key can never reach the fetch layer and get echoed by undici's header validation error. admin-billing requestAdminJson additionally wraps the fetch call and scrubs key-shaped tokens from any transport error as defense-in-depth. - [P2] Fail-loud pagination: the Anthropic, OpenAI, and Claude Code analytics clients no longer treat has_more:true with a null cursor as a clean end-of-pages. They now throw rather than silently returning a partial (understated) bill / usage picture. - [P2] Honest key verification: thread a "was this vendor actually queried" signal worker → service → renderer so the Cost tab reports "Saved and verified" only when local usage really exercised the key, and an honest "no recent local usage to verify against yet" otherwise. Testing: - pnpm -C apps/desktop typecheck — clean - pnpm -C apps/desktop lint — clean - pnpm -C apps/desktop test — 2041 pass / 0 fail (added regression tests for each finding: control-char key rejection, has_more-without-cursor throws for all three clients, and the queried-vendor signal across worker/service/renderer). Risks: - Low. Boundary guard is additive and rejects only keys that could never have authenticated anyway. Pagination change converts a silent undercount into a loud failure on a malformed vendor response. The verification signal is an internal IPC field (ships as one unit, no migration). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
Codex PR review round — addressed in
|
Contributor
Author
Prototype handoff → PRD-426This PR is a working prototype, not a production candidate. It has been handed off to product + engineering under a new parent PRD: PRD-426 — 004. Token cost & reconciliation: prototype → production Key points for anyone picking this up:
Architecture validated here that should carry forward: |
…st priced cells When every local model is unpriced (e.g. a new model before genai-prices adds it), aggregateLocal() produces no cells and the vendor APIs were never queried. Now the window is derived from all metered usage rows so vendor-billed costs still surface even with zero local estimate. Addresses Codex P2: "Base reconciliation windows on raw usage days." Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
Addressed the Codex P2 in 87f1f39: The reconciliation window is now derived from all metered usage rows (via - const localDays = [...local.values()].map((cell) => cell.day);
- const window = computeWindow(localDays);
+ const allUsageDays = usageRows
+ .filter((r) => isMeteredApi(r.billingMode))
+ .map((r) => { const d = parseDate(r.startedAt); return d ? toUtcDay(d) : null; })
+ .filter((d): d is string => d !== null);
+ const window = computeWindow(allUsageDays); |
…amp conflicts - package.json: keep branch version 0.15.103 (higher than main's 0.15.101) - build-agent-monitor.mjs: keep branch's full overlay stamp set (Dashboard + Settings + Ledger) with FEA-1434 cache-bust comment Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compatibility Smoke Test Results
|
The test previously asserted vendor APIs were NOT called for unpriced models. Now that the window is derived from all metered usage rows (not just priced cells), vendor APIs are correctly called so vendor-billed costs surface even for unknown models. Update the assertion to match the intended behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mikeangstadt
approved these changes
Jun 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rebuilds the desktop app's token-cost accounting on
@pydantic/genai-pricesas the single source of truth, and layers vendor cost reconciliation + drift diagnostics on top. Delivered as one PR spanning five tickets (FEA-1431/1433/1434/1435/1436). 52 files, ~9.7k insertions, 20 commits.Core principle throughout: trust the pricing library. Nothing overrides, clamps, asserts-away, or rewrites what genai-prices returns. There is no silent
$0— an unpriced model surfaces as unpriced, never as a fabricated zero.What ships
@pydantic/genai-prices(0.0.62) in desktop-main (src/shared/token-cost.ts) and wires it into the agent-monitor sidecar.billing_mode(subscription vs API) stamped on all write paths, schema migration + sync-contract propagation, and a split metered-vs-subscription ledger surfaced in the UI./v1/organizations/cost_report) + OpenAI (/v1/organization/costs) billing clients, integer micro-cent money math + drift computation, a nightly reconciliation worker, a persisted reconciliation store, and a Settings sub-tab (Admin keys + drift table)./v1/organizations/usage_report/claude_code), aggregated per user. Clearly labeled as Anthropic's estimate; it never feeds or overrides the local genai-prices ledger.Architecture & security notes
Number.isSafeIntegerdiscipline (no floats, no BigInt). Non-2xx vendor responses and page-cap overruns throw rather than returning an understated/partial bill.Independent pre-PR review
Two independent reviewers audited (A: Claude Code analytics slice; B: reconciliation + admin-key + money-math core). Both returned SHIP WITH MINOR FIXES — no Critical/High. All actionable findings were addressed in the final commit (key-redaction in vendor error bodies, non-negative token guard, impossible-calendar-date rejection, renderer
window-shadow + label fixes), each with a regression test asserting the exact invariant. One finding (readOnly: trueon the dashboard.db handle) was deliberately declined with documented rationale: aSQLITE_OPEN_READONLYconnection cannot attach to the sidecar's live WAL database without an existing writer, so it would intermittently fail to open a readable file; the handle is read-write by design and issues only SELECTs.Test plan
pnpm typecheck— cleanpnpm lint— cleanpnpm test— 2035 pass / 0 fail (1931 main + 104 scripts)node:sqlitefrom the asar-external universal binary)🤖 Generated with Claude Code