feat(sse): honor provider-rule lock scope for agentrouter (connection vs model) (#10419)

Makes the ProviderErrorRule `scope` field real at the persistence layer, exclusively for agentrouter (owner decision; every other provider keeps byte-identical behavior).

checkFallbackError now surfaces `ruleScope` behind the HONORS_RULE_LOCK_SCOPE_PROVIDERS allowlist, and the agentrouter 403 path consults the rules before the generic apikey-FORBIDDEN early-return. markAccountUnavailable honors scope "connection" with a temporary connection cooldown instead of a per-model lockout — guarded so a permanent state can never be downgraded to a transient retry loop — and combo now skips the exhausted account within the same request, which also stops force-reusing the just-cooled connection via allowRateLimitedConnection.

Documented in RESILIENCE_GUIDE §7 with the honest limits (disableCooling connections keep per-model behavior; the 6h model-access cooldown is clamped by mlSettings.maxCooldownMs, 30min by default; same-request skip needs targets carrying their own connectionId).

Closes #10334

Author: diegosouzapw

docs/architecture/RESILIENCE_GUIDE.md

@@ -330,32 +330,75 @@ excludeMarkers, defaultRetryAfterMs}`), matched via `applyStatusRestatement()`.
 
 Permanent errors (agentrouter's `无权访问模型` — no access to this model) are
 NEVER restated: `excludeMarkers` vetoes the rule even when `textMarkers` hit,
-so the error keeps its original status and nothing retries it forever. A
-separate provider classification rule
-(`agentrouter-model-access-denied` in `open-sse/config/providerErrorRules.ts`)
-declares an `auth_error`/scope-`model` match for this text, but it does not
-fire on the live production path today: the rule only matches `status ===
-403`, and `checkFallbackError`'s apikey-category `FORBIDDEN` branch
-(`open-sse/services/accountFallback.ts`) returns early for a plain 403
-*before* the provider-rule lookup ever runs. In practice a `无权访问模型` 403
-is handled the same way as the base apikey-provider 403 path (see Connection
-Cooldown, §2), not as a 6h model lockout. The rule still exists as a
-declarative classification consumable by future callers of `classifyError`
-with context — wiring it into the production `checkFallbackError` path is
-tracked as a follow-up, not yet done.
-
-Restated quota errors (`额度不足`) do reach a provider rule in production
-(`agentrouter-user-quota-exhausted`, scope `"connection"`), but `scope` on
-`ProviderErrorRuleMatch` is currently informational — the persistence path
-(`checkFallbackError` → `combo.ts`) only consumes `reason` and `cooldownMs`,
-never `scope`. What actually happens for agentrouter (`passthroughModels:
-true` → `hasPerModelQuota()` returns `true`) is a **per-model** lockout via
-`recordModelLockoutFailure()`: the connection itself is never cooled down for
-this error (`combo.ts` skips `recordProviderCooldown` for 429 when
-`hasPerModelQuota` is true), so other models on the same account keep being
-tried — each one burns one call and its own lockout before combo routing
-moves on. Honoring `scope` end-to-end (so a `"connection"` match actually
-locks the connection) is tracked as a follow-up.
+so the error keeps its original status and nothing retries it forever. The
+matching provider classification rule
+(`agentrouter-model-access-denied` in `open-sse/config/providerErrorRules.ts`:
+`reason: "auth_error"`, `scope: "model"`, a `6h` declared base cooldown) is
+consulted by `checkFallbackError` (`open-sse/services/accountFallback.ts`)
+*before* the generic apikey-category `FORBIDDEN` early-return, gated on
+`honorsRuleLockScope(provider)` (#10334 — currently agentrouter-exclusive via
+the `HONORS_RULE_LOCK_SCOPE_PROVIDERS` allowlist in
+`providerErrorRules.ts`). The rule's declared 6h cooldown flows through as
+`fallbackResult.baseCooldownMs`, but it still feeds the pre-existing
+per-model-quota lockout path (`lockModelIfPerModelQuota()` /
+`recordModelLockoutFailure()`, unchanged by #10334 except for the cooldown
+source): it is clamped down to the operator's `mlSettings.maxCooldownMs`
+(default `1_800_000ms` / 30min), like every other model lockout, and the
+*persisted lockout reason* stays the pre-existing hardcoded `"forbidden"`,
+not the rule's `"auth_error"` — only the cooldown duration is honored
+end-to-end, not the reason string. The connection itself stays active;
+sibling models on the same connection are unaffected.
+
+Restated quota errors (`额度不足`) reach a provider rule in production
+(`agentrouter-user-quota-exhausted`: `reason: "quota_exhausted"`, `scope:
+"connection"`, no declared cooldown of its own — the persistence layer's
+scaled backoff default applies). Since #10334, `scope` on
+`ProviderErrorRuleMatch` IS consumed end-to-end, but **only** for providers in
+the `HONORS_RULE_LOCK_SCOPE_PROVIDERS` allowlist (`providerErrorRules.ts` —
+today only `"agentrouter"`, gated via `honorsRuleLockScope()`). For every
+other provider `scope` remains informational, exactly as before #10334.
+`checkFallbackError` surfaces the matched rule's scope as
+`fallbackResult.ruleScope`; `isAgentrouterConnectionQuotaScope()`
+(`src/sse/services/auth.ts`) is the shared guard that confirms a
+`ruleScope` is genuinely safe to honor as a connection-wide, self-recovering
+signal (scope `"connection"`, reason `quota_exhausted`, never `permanent`,
+never `creditsExhausted` — a defense against a future rule pairing scope
+`"connection"` with a permanent account state). Two consumers call it:
+
+- **Persistence** (`markAccountUnavailable()`, `src/sse/services/auth.ts`):
+  instead of falling into the passthrough-provider **per-model** lockout
+  branch (agentrouter is `passthroughModels: true` → `hasPerModelQuota()`
+  returns `true`), it applies a **temporary connection cooldown** —
+  `testStatus: "unavailable"` + `rateLimitedUntil`, never a terminal status
+  (`credits_exhausted`/`banned`/`expired`) — so the connection self-recovers
+  once the cooldown lapses instead of requiring a manual credential reset.
+  Skipped for connections with `disableCooling: true` (#2997): that opt-out
+  falls through to the per-model lockout instead (a documented trade-off —
+  see the code comment above the branch).
+- **Same-request combo routing** (`applyComboTargetExhaustion()`,
+  `open-sse/services/combo/targetExhaustion.ts`): the same guard marks the
+  connection into the in-memory `exhaustedConnections` set, keyed
+  `${provider}:${connectionId}`. This only skips a remaining SAME-REQUEST
+  target that *itself already carries that exact `connectionId`* on its own
+  target object (`getExhaustedTargetSkipReason()`,
+  `open-sse/services/combo/comboPredicates.ts`, `if (provider &&
+connectionId)` before the `exhaustedConnections` lookup) — a plain
+  model-list combo, where sibling targets carry no pinned `connectionId` of
+  their own and one is only resolved per-dispatch from the response's
+  `X-OmniRoute-Selected-Connection-Id` header, never hits that key match. For
+  that common case, the real protection against a remaining leg reusing the
+  just-exhausted account is NOT this Set — it is the persistence layer above
+  (the connection's `rateLimitedUntil` is now in the future) combined with
+  this same guard suppressing `transientRateLimitedProviders` for the
+  failure (see "Two-stage design" and the code comment on the
+  `isAgentrouterConnectionQuotaScope` branch in `targetExhaustion.ts`): with
+  that Set left unmarked, `combo.ts`'s `allowRateLimitedConnection` force-allow
+  (`open-sse/services/combo.ts:1005-1013`, `:2734-2738`) does NOT kick in for
+  the provider's remaining legs, so credential selection's `rateLimitedUntil`
+  filter (`src/sse/services/auth.ts:1238`) is honored normally and a
+  remaining leg either picks a different, still-eligible agentrouter
+  connection or fails with no credentials available — it does not force its
+  way back onto the connection this branch just cooled down.
 
 ### Two-stage design: status restatement, then classification
 
@@ -380,6 +423,15 @@ allowlisted providers, the structured error otherwise. Adding a provider to
 that the default path for every provider not on the list stays
 byte-for-byte unchanged.
 
+A rule's `scope` (`model` / `provider` / `connection`) is a separate opt-in
+from `FULL_TEXT_RULE_PROVIDERS`: `checkFallbackError` only surfaces it as
+`fallbackResult.ruleScope`, and downstream consumers only honor it as
+anything other than an informational label, for providers in the
+`HONORS_RULE_LOCK_SCOPE_PROVIDERS` allowlist in the same file (`gated via
+honorsRuleLockScope()` — today only `"agentrouter"`). See "Restated quota
+errors" above for what a `scope: "connection"` match actually does once a
+provider is on that allowlist.
+
 ### Adding a new quota-misstating gateway
 
 1. Register one rule array in `statusRestatementRegistry`
@@ -395,7 +447,15 @@ byte-for-byte unchanged.
    `checkFallbackError` only ever hands the rule the structured
    `{code, type}` error and a body-text rule will never match live traffic.
    Rules that match purely on `status`/`headers` (like Opencode's or
-   Minimax's) do not need this opt-in.
+   Minimax's) do not need this opt-in. Separately, if the rule declares
+   `scope: "connection"` and the intent is an actual connection-wide cooldown
+   plus same-request combo skip (not just an informational label), add the
+   provider id to `HONORS_RULE_LOCK_SCOPE_PROVIDERS` in the same file — this
+   is what gates `isAgentrouterConnectionQuotaScope()`-style consumption in
+   `markAccountUnavailable()` (`src/sse/services/auth.ts`) and
+   `applyComboTargetExhaustion()`
+   (`open-sse/services/combo/targetExhaustion.ts`); without it, `scope`
+   still flows through `fallbackResult.ruleScope` but nothing acts on it.
 3. Add unit tests mirroring `tests/unit/upstream-status-restatement.test.ts`
    and `tests/unit/agentrouter-error-rules.test.ts` (including the
    not-permanent / not-creditsExhausted guards, and — if the provider needs

open-sse/config/providerErrorRules.ts

@@ -30,13 +30,15 @@ export type ProviderErrorRule = {
 export type ProviderErrorRuleMatch = {
   reason: ConfiguredErrorReason;
   /**
-   * Intended lock scope. NOTE: this field is currently INFORMATIONAL — no
-   * consumer of `getProviderErrorRuleMatch` (checkFallbackError, combo.ts)
-   * reads `scope` today; only `reason` and `cooldownMs` are consulted. The
-   * actual lock scope applied at runtime is decided independently by each
-   * call site (e.g. `hasPerModelQuota()` deciding model- vs connection-level
-   * lockout). Honoring this field end-to-end is tracked as a follow-up —
-   * see `docs/architecture/RESILIENCE_GUIDE.md` §7.
+   * Intended lock scope. #10334: this field is CONSUMED end-to-end only for
+   * providers in `HONORS_RULE_LOCK_SCOPE_PROVIDERS` (agentrouter-exclusive
+   * today, gated by `honorsRuleLockScope()`) — for those, `checkFallbackError`
+   * surfaces it as `ruleScope` on its return value for the persistence layer
+   * to honor instead of re-deriving scope from `hasPerModelQuota()`. For
+   * every other provider it remains INFORMATIONAL: `getProviderErrorRuleMatch`
+   * callers still read only `reason`/`cooldownMs`, and the actual lock scope
+   * is decided independently by each call site. Widening the allowlist is
+   * tracked as a follow-up — see `docs/architecture/RESILIENCE_GUIDE.md` §7.
    */
   scope: "model" | "provider" | "connection";
   /** Optional explicit cooldown; falls back to the existing per-reason defaults. */
@@ -188,31 +190,29 @@ function buildOpenrouterRules(): ProviderErrorRule[] {
 // agentrouter.org misstates temporary quota exhaustion as 403/400 with a
 // Chinese body. upstreamStatusRestatement.ts rewrites the status to 429
 // BEFORE classification, so rules here accept both the raw 403/400 and the
-// restated 429 (text is the real discriminator either way). In production,
-// the raw 403 path is what actually matters here: checkFallbackError's
-// apikey-category FORBIDDEN branch (~line 1699) returns EARLY for a plain
-// 403, before these rules are ever consulted — these rules fire on the
-// RESTATED 429 (chatCore's upstreamStatusRestatement hook runs first) via
-// resolveRuleMatchBody, which is the only path in checkFallbackError that
-// hands these rules the full error text instead of just {code, type}.
+// restated 429 (text is the real discriminator either way). Both the raw 403
+// path AND the restated 429 path reach these rules in production:
+// checkFallbackError's `honorsRuleLockScope("agentrouter")` pre-check
+// (#10334) consults these rules BEFORE the generic apikey-category FORBIDDEN
+// branch, and the restated 429 reaches them via the existing provider-rule
+// lookup in the configured-rule branch. Both paths use resolveRuleMatchBody,
+// the only mechanism in checkFallbackError that hands agentrouter's rules the
+// full error text instead of just {code, type}.
 //  - "额度不足": account-wide temporary quota → quota_exhausted, scope
 //    "connection" (mirror of the Opencode account-wide rationale above).
-//    NOTE: `scope` on ProviderErrorRuleMatch is currently informational —
-//    checkFallbackError/combo.ts only consume `reason` and `cooldownMs`, not
-//    `scope`. For agentrouter specifically (passthroughModels: true →
-//    hasPerModelQuota() is true), this quota_exhausted match actually
-//    resolves to a PER-MODEL lockout (recordModelLockoutFailure), not a
-//    connection-wide lock — other models on the same account keep being
-//    tried by combo routing (each burning one call) until they lock out
-//    individually. Honoring `scope` end-to-end is tracked as a follow-up.
+//    `scope` on ProviderErrorRuleMatch is CONSUMED for agentrouter (#10334,
+//    exclusive allowlist via `honorsRuleLockScope`): checkFallbackError
+//    surfaces it as `ruleScope` on its return value. Whether the persistence
+//    layer (markAccountUnavailable / combo target exhaustion) actually
+//    switches from `hasPerModelQuota()`-derived scope to honoring `ruleScope`
+//    is Tasks 2/3 of #10334 — this task only surfaces the field.
 //  - "无权访问模型": declares auth_error/scope "model" (intent: lock only the
 //    model so the connection keeps serving the rest — Model Lockout tier).
-//    This rule does NOT fire on the production path today: it only matches
-//    `status === 403`, but checkFallbackError's apikey FORBIDDEN branch
-//    returns early for a plain 403 before this rule is ever consulted (see
-//    the note above). A live `无权访问模型` 403 is handled like the base
-//    apikey-provider 403 today. Wiring this rule into that path is tracked
-//    as a follow-up.
+//    This rule now fires on the production 403 path (#10334): the
+//    `honorsRuleLockScope` pre-check matches it and returns its declared
+//    reason/cooldown/scope before the generic apikey-FORBIDDEN early-return
+//    ever runs. A live `无权访问模型` 403 therefore no longer falls through to
+//    the base apikey-provider 403 handling.
 function buildAgentrouterRules(): ProviderErrorRule[] {
   const AGENTROUTER_ERROR_STATUSES = new Set([400, 403, 429]);
   return [
@@ -231,8 +231,15 @@ function buildAgentrouterRules(): ProviderErrorRule[] {
         if (status !== 403) return null;
         const text = JSON.stringify(body ?? "").toLowerCase();
         if (!text.includes("无权访问模型")) return null;
-        // 6h: effectively "until the operator fixes the key's model grants",
-        // without being an unrecoverable terminal state.
+        // Declares a 6h cooldown, but the effective cooldown is NOT 6h: the
+        // model-lockout persistence layer (recordModelLockoutFailure, called from
+        // markAccountUnavailable) clamps every base cooldown — this one included —
+        // to the configured model-lockout maxCooldownMs, which defaults to
+        // 1_800_000ms / 30min (src/lib/resilience/modelLockoutSettings.ts,
+        // DEFAULT_MODEL_LOCKOUT_SETTINGS.maxCooldownMs). So in practice this is
+        // "locked for ~30min by default (up to 6h if an operator raises the model-
+        // lockout cap in settings)", not "until the operator fixes the key's model
+        // grants" — it is a recoverable window, not a real fix-driven unlock.
         return { reason: "auth_error", scope: "model", cooldownMs: 6 * 60 * 60 * 1000 };
       },
     },
@@ -255,6 +262,21 @@ export const providerRuleRegistry = new Map<string, ProviderErrorRule[]>([
   ["agentrouter", buildAgentrouterRules()],
 ]);
 
+/**
+ * Providers whose ProviderErrorRuleMatch.scope is actually CONSUMED at the
+ * persistence layer (markAccountUnavailable / combo target exhaustion) to pick
+ * connection-vs-model lock scope. EXCLUSIVE allowlist by owner decision
+ * (2026-08-14, issue #10334) — deliberately SEPARATE from
+ * FULL_TEXT_RULE_PROVIDERS: that set controls what body a rule matches against
+ * (input), this one controls whether the matched scope changes caller behavior
+ * (output). A provider could need one without the other.
+ */
+const HONORS_RULE_LOCK_SCOPE_PROVIDERS = new Set(["agentrouter"]);
+
+export function honorsRuleLockScope(provider: string | null | undefined): boolean {
+  return !!provider && HONORS_RULE_LOCK_SCOPE_PROVIDERS.has(provider.toLowerCase());
+}
+
 /**
  * Providers whose rules match on the FULL upstream error text.
  * checkFallbackError's rule lookup normally passes only the structured