Firestore: assertion ca9 (pendingResponses < 0) still reachable in 12.17.0 after #9842, and the watch stream keeps pumping into the failed AsyncQueue (b815 flood → tab crash)

[islamokasha1-cloud]

Operating System

Windows 10/11 (Chrome 151), Android 10 (Chrome 149), iOS 17.1 (Mobile Safari 17.1) — reproduces on all three

Environment (if applicable)

Browser (Chrome 151.0.0.0 on Windows, Chrome 149 on Android, Mobile Safari 17.1 on iOS)

Firebase SDK Version

12.17.0 (compat build, loaded from https://www.gstatic.com/firebasejs/12.17.0/ — bundles @firebase/firestore 4.17.0)

Firebase SDK Product(s)

Firestore

Project Tooling

No bundler. Plain single-page app; compat SDK loaded via <script> tags from gstatic.
Firestore configured as:
  db.settings({ experimentalForceLongPolling: true })
  no enablePersistence() — memory cache only, no IndexedDB, no multi-tab sync
App Check: ReCaptchaEnterpriseProvider, enabled before any Firestore call.

Detailed Problem Description

We are hitting the assertion that #9842 was supposed to fix, on a version that already
contains that fix. Once it fires, the app is permanently dead until a full page reload,
and the tab then floods with uncaught errors until the browser kills it.
What the user sees
An open page (a dashboard with ~10 active onSnapshot listeners) stops updating. Every
loading placeholder stays on screen forever. The console shows one uncaught error per
second, ~985 of them before the user gives up and reloads. No listener ever fires again.
The error

Uncaught Error: FIRESTORE (12.17.0) INTERNAL ASSERTION FAILED: Unexpected state (ID: b815)
CONTEXT: {"rl":"Error: FIRESTORE (12.17.0) INTERNAL ASSERTION FAILED: Unexpected state
(ID: ca9) CONTEXT: {\"M\":-1,\"targetId\":1002}\n
    at Rt (https://www.gstatic.com/firebasejs/12.17.0/firebase-firestore-compat.js:1:200577)
    at f  (…:1:200665)
    at fi.J (…:1:252100)
    at (…:1:252585)
    at Array.forEach (<anonymous>)
    at Di.forEachTarget (…:1:252975)
    at Di.ae (…:1:252471)
    at Cd (…:7:2746)
    at Ra.onNext (…:1:283128)
    at (…:1:280929)"}
    at Rt (assert.ts:89:9)
    at T (assert.ts:63:3)
    at Bp.Xc (async_queue_impl.ts:239:7)
    at Bp.enqueue (async_queue_impl.ts:120:10)
    at Bp.enqueueAndForget (async_queue_impl.ts:94:10)
    at persistent_stream.ts:592:18
    at Na.Tt (persistent_stream.ts:531:7)
    at Na.At (stream_bridge.ts:94:10)
    at webchannel_connection.ts:432:26
    at b.<anonymous> (webchannel_connection.ts:70:7)

Reading the two IDs against the source (I read the repo at commit 317dcd9):
0x0ca9 is the hardAssert in TargetState.recordTargetResponse()
(packages/firestore/src/remote/watch_change.ts:247-254):
``pendingResponses is less than 0. This indicates that the SDK received more target acks from the server than expected. The SDK should not continue to operate.
The reported context `{"M":-1,"targetId":1002}` is
`{ pendingResponses: -1, targetId: 1002 }`.
`0xb815` is `AsyncQueueImpl.verifyNotFailed()`
(`packages/firestore/src/util/async_queue_impl.ts:239`): `'AsyncQueue is already failed'`.
So there are two distinct problems here.

Problem A — ca9 is still reachable on 12.17.0, i.e. after #9842
402b1f0 / #9842 ("Assertion ID: ca9 (pendingResponses less than 0) caused by target
creation race condition") shipped in @firebase/firestore@4.14.1 = firebase@12.13.0.
85f6f4e / #9985 ("improved robustness and logging in query listen stream creation and
re-creation") is also in. I verified both are ancestors of the 12.17.0 release commit
(d459c97). We run 12.17.0, which bundles @firebase/firestore@4.17.0 — so both
fixes are present and the assertion still fires.
Notably we are not in the configurations usually blamed for this class of bug:
experimentalForceLongPolling: true (so this is not WebChannel streaming-specific)
no IndexedDB persistence — memory cache only
no multi-tab synchronization
single tab, single FirebaseApp, single Firestore instance, one copy of the SDK
The targetId: 1002 is worth noting: this is a long-lived page that allocates many
transient listen targets over a session (a large number of one-shot get() calls
alongside ~10 long-lived onSnapshot listeners). Frequency correlates with session
length rather than with any single user action, which is what you would expect from a
race in target allocation / re-listen rather than from one specific query.
Problem B — after the fatal assertion, the stream keeps delivering into the dead queue
This is the part that turns a single internal error into a browser-tab crash, and I think
it is independently fixable regardless of Problem A.
The ca9 assertion kills the AsyncQueue (that is its stated intent: "The SDK should not
continue to operate"). But the WebChannel/watch stream is not torn down. Per the stack
above, every subsequent message still arrives at webchannel_connection.ts →
stream_bridge.ts → persistent_stream.ts:592 → enqueueAndForget() →
verifyNotFailed() → fail(0xb815) — uncaught, roughly once per second, unbounded.
Consequences for an app in production:
Unbounded uncaught errors. On long-lived tabs we have seen the renderer die outright
(Chrome STATUS_ACCESS_VIOLATION, "Aw, Snap"), which loses unsaved user input.
There is no supported recovery. terminate() cannot help, because terminate() itself
enqueues onto the failed queue. clearPersistence() likewise. So a full page reload
is the only recovery — from an app's point of view, one internal assertion means the
whole Firestore client is unrecoverable for the lifetime of the document.
It is not silenceable. Because these surface as uncaught errors on window, any global
error reporter records hundreds of duplicates of a single fault.
Requests for Problem B:
When the AsyncQueue enters the failed state, close the watch/write streams (and stop
the WebChannel) instead of leaving them delivering messages into a queue that can only
throw.
And/or make enqueueAndForget() a no-op once the queue has failed — the "forget"
variant has no caller waiting on a result, so converting a guaranteed uncaught throw
into a single logged warning loses nothing and stops the flood.
Consider making the failed state observable to the application (e.g. surface it through
onSnapshotsInSync error handling or a dedicated event), so an app can tell the user
"reload to resume" instead of showing a spinner that will never resolve. Today an app
can only detect this by string-matching "INTERNAL ASSERTION FAILED" on window.onerror,
which is what we had to implement.

A question about sendWatchRequest (observation from reading the source, not a proven cause)
In packages/firestore/src/remote/remote_store.ts:462, sendWatchRequest() calls
recordPendingTargetRequest(targetId) first, and then may return early without ever
calling .watch(...):

function sendWatchRequest(remoteStoreImpl, remoteTargetData): void {
  remoteStoreImpl.watchChangeAggregator!.recordPendingTargetRequest(
    remoteTargetData.targetId
  );

  if (resumeToken.approximateByteSize() > 0 || snapshotVersion > SnapshotVersion.min()) {
    const sdkTargetId = remoteStoreImpl.targetIdMapRemoteToSdk.get(/* … */);
    if (sdkTargetId === undefined) {
      logDebug(LOG_TAG, 'SDK target ID not found for remote ID: ' + remoteTargetData.targetId);
      // There's already a new remoteStoreListen request for the original target, so ignore this.
      return;                       // ← pending recorded, but no watch request sent
    }
    // …
  }

  ensureWatchStream(remoteStoreImpl).watch(remoteTargetData);
}

On that path pendingResponses is incremented for a request that is never sent, so no ack
can ever balance it and TargetState.isPending stays true for that target. Is that
intended? The sign is the opposite of ca9 (count too high, not negative), so it is
probably a different symptom — a target that stays permanently pending — but it is in the
same accounting path, and a target whose snapshot is never raised would look to an app
exactly like the "spinner that never resolves" we are chasing.

Steps and code to reproduce issue

I do not have a minimal reproduction, and I want to be straightforward about that. The
failure is a race: it appears after a session has been open for a while (minutes to hours),
across different users, browsers and operating systems, and never on a specific user action
we can point to. What follows is the shape of the app and everything we can pin down.
Setup:

firebase.initializeApp(firebaseConfig);
firebase.appCheck().activate(
  new firebase.appCheck.ReCaptchaEnterpriseProvider(SITE_KEY), true);

const db = firebase.firestore();
db.settings({ experimentalForceLongPolling: true });
// deliberately no enablePersistence() — memory cache only

Load shape:
~10 long-lived collection listeners, attached once after sign-in and kept for the
lifetime of the page, e.g.:

  db.collection('global_purchases').orderBy('createdAt', 'desc').limit(400)
    .onSnapshot(snap => { /* re-render */ }, err => console.warn(err));

Others use .limit(500), .limit(300), plain .orderBy(...), and single-document
db.doc(path).onSnapshot(...).
Alongside these, a large number of one-shot db.doc(...).get() / collection().get()
calls during boot and normal use. Over a session, allocated target IDs climb into the
high hundreds (targetId: 1002 in the report above).
Writes are ordinary set(..., { merge: true }) / update() / occasional batch(), some
issued while listeners are live.
Auth is a custom token; the page also signs in anonymously in some flows. Auth state can
change while listeners are attached.
What we have already ruled out on our side (each of these was tried in production over
several releases, and the assertion survived all of them):
multi-tab synchronization — disabled
IndexedDB persistence — removed entirely (memory cache only)
listener churn — global listeners are now attached exactly once and never detached until
sign-out; they are no longer torn down and re-attached on navigation
long-polling — forced
Frequency: intermittent; on a busy day, several times across our user base. When it
happens, that page is dead until reload.
If it would help, I can add temporary instrumentation around
recordPendingTargetRequest / recordTargetResponse (target ID, resume-token presence,
and whether the target was just re-listened) on a patched build and report what precedes
the negative count. Tell me what you would find most useful and I will collect it.