fix(workspace, transport): invokes survive reconnects — the claim protocol

Twice in one evening a crawl finished server-side while the tab spun
forever, both times severed by a routine workspace-service rebuild. Root
cause: the transport rejected and CLEARED its pending invokes on every
socket close, and the server had nowhere to put a result whose socket
had died.

Client: pending invokes now survive the drop — undelivered queued frames
re-send on reconnect; delivered ones re-attach via a new claim frame per
pending id, answered through the normal result path. Chat turns stay
fail-fast. Server: every invoke's serialized result is tracked
in-flight and stashed (15-min TTL) when its socket is gone; claims
return the stash, attach to running work, or fail fast with an explicit
"service restarted; retry" when the process lost the maps. The 660s
client deadline stays as backstop; durable-across-restart results park
with the liveness sweep (#21).

Closes #41.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvYzx7vDWeafnkAi2nEQeb

Author: mpstaton

context-v/issues/Invokes-Survive-Reconnects-The-Claim-Protocol.md

@@ -0,0 +1,65 @@
+---
+title: "Invokes survive reconnects — the claim protocol closes the eternal-spinner root cause"
+lede: "Twice in one evening a crawl finished server-side while the tab spun forever — both times because a workspace-service rebuild severed the socket and the transport threw away its pending invokes. Pending invokes now survive the drop and re-attach by id: the server stashes results whose socket died, and reconnecting clients send claim frames to collect them."
+date_created: 2026-07-24
+date_modified: 2026-07-24
+authors:
+  - Michael Staton
+augmented_with:
+  - Claude Code on Claude Fable 5
+semantic_version: 0.0.0.1
+tags:
+  - Issue
+  - Augment-It
+  - Workspace
+  - Transport
+  - Reliability
+  - Didi-Crawl
+status: Shipped
+date_first_published: 2026-07-24
+post_ship_note: "Shipped 2026-07-24 — successor to [[Crawl-Replies-Can-Be-Lost-Eternal-Spinner-No-Client-Timeout]]'s mitigations, landing the root cure the same evening after the Atlas Network crawl reproduced the loss. gh #41 closed."
+---
+
+# The claim protocol
+
+## Why (twice in one evening)
+
+The Curry Foundation crawl completed in 87s server-side; the tab spun
+forever. Mitigations landed (client deadline, bigger dispatch ceiling) — and
+then the Atlas Network crawl reproduced the loss within the hour, this time
+severed by the very rebuild that shipped the mitigation. Container rebuilds
+are ROUTINE in this stack; pending invokes must survive them.
+
+## The design
+
+- **Client** (`packages/workspace/src/transport.ts`): on socket close,
+  pending invokes are NOT rejected (previously: reject + clear — the root
+  cause). Undelivered queued frames re-send on reconnect; delivered ones
+  re-attach by sending a `claim` frame per pending id. The claim's answer is
+  a normal result frame — same id, same resolution path. Chat turns stay
+  fail-fast (cheap to resend).
+- **Server** (`services/workspace/src/ws.ts`): every invoke's serialized
+  result is tracked in-flight; if the owning socket is gone when the result
+  lands, it's stashed (15-min TTL). A `claim` returns the stash, attaches to
+  still-running work, or — when the process restarted and knows nothing —
+  replies with an explicit "workspace service restarted; retry" error so the
+  caller fails fast instead of hanging.
+
+## What this covers, and what it doesn't
+
+Covered: network blips, browser sleep/wake, idle disconnects, and
+mid-dispatch workspace-service RESTARTS where the service comes back before
+the client gives up (in-flight work is lost with the process, but the claim
+gets an immediate explicit error). Not covered: durable results across
+restarts — the responder services (prompt-runner) reply over core NATS to a
+requestor that no longer exists; true durability needs persisted results or
+JetStream-style delivery, parked with the liveness sweep
+([[Live-Not-Live-Indicator-Tooling-And-Cross-Service-Error-Surfacing]]).
+The 660s client deadline from the mitigation pass stays as the backstop.
+
+## Rider observation (operator, same session)
+
+`services/workspace/src/ws.ts` is a poor name — terse, collides mentally
+with both "workspace" and the `ws` npm package. `frame-router.ts` or
+`websocket-router.ts` would say what it is. Not renamed mid-fix (drift
+policy); a candidate for the conventions/component cleanup (#22-adjacent).

packages/workspace/src/transport.ts

@@ -92,7 +92,21 @@ export function createTransport(config: TransportConfig): Transport {
     ws.addEventListener('open', () => {
       backoff = RECONNECT_INITIAL_MS;
       config.onStatus?.('open');
+      // Frames still in the queue were never delivered — flush re-sends
+      // them as ordinary invokes. Pending entries NOT in the queue were
+      // delivered before a drop: re-attach to their (possibly finished)
+      // server-side work with claim frames (gh #41). The server answers a
+      // claim with the normal result frame — same id, same resolution path
+      // — or an explicit not-found error if it restarted meanwhile.
+      const queuedIds = new Set(
+        sendQueue.filter((fr) => fr.kind === 'invoke').map((fr) => fr.id),
+      );
       flushSendQueue();
+      for (const id of pending.keys()) {
+        if (!queuedIds.has(id)) {
+          ws!.send(JSON.stringify({ kind: 'claim', id }));
+        }
+      }
     });
 
     ws.addEventListener('message', (evt: MessageEvent) => {
@@ -137,17 +151,24 @@ export function createTransport(config: TransportConfig): Transport {
 
     ws.addEventListener('close', () => {
       config.onStatus?.('closed');
-      // Reject any in-flight invokes (already on the wire or still
-      // queued) — callers will receive a clean 'socket closed' and can
-      // retry. Clearing the sendQueue too keeps pending and queue in
-      // lockstep; a stray queued frame surviving a reconnect would
-      // produce a server reply that no longer has a pending entry to
-      // resolve, wasting server work.
-      for (const [, p] of pending) p.reject(new Error('socket closed'));
-      pending.clear();
+      // In-flight INVOKES survive the drop (gh #41): their pending entries
+      // stay put and the next 'open' re-attaches via claim frames — the
+      // server holds results for invokes whose socket died. Long-running
+      // work (didi crawls run minutes) no longer strands an eternal
+      // spinner because a container rebuild or network blip severed the
+      // socket. Chat turns stay fail-fast: cheap to resend, and the rail
+      // shows the error inline.
+      if (closing) {
+        for (const [, p] of pending) p.reject(new Error('socket closed'));
+        pending.clear();
+      }
       for (const [, p] of chatPending) p.reject(new Error('socket closed'));
       chatPending.clear();
+      // Keep undelivered invoke frames for re-send on reconnect; drop
+      // queued chat frames (their pending entries were just rejected).
+      const keep = sendQueue.filter((fr) => fr.kind === 'invoke' && pending.has(fr.id));
       sendQueue.length = 0;
+      sendQueue.push(...keep);
       if (!closing) scheduleReconnect();
     });
 

packages/workspace/src/types.ts

@@ -309,8 +309,13 @@ export type ChatErrorFrame = {
   error: string;
 };
 
+// Re-attach to an invoke after a reconnect (gh #41) — the server replies
+// with the normal ResultFrame for that id (possibly stashed from before the
+// drop), or ok:false when it restarted and no longer knows the invoke.
+export type ClaimFrame = { kind: 'claim'; id: string };
+
 export type ServerFrame = ResultFrame | EventFrame | SessionFrame | ChatResponseFrame | ChatErrorFrame;
-export type ClientFrame = InvokeFrame | ChatTurnFrame;
+export type ClientFrame = InvokeFrame | ChatTurnFrame | ClaimFrame;
 
 // --- request-reviewer / response-reviewer surfaces ---
 // See context-v/specs/Request-Reviewer-Pre-Flight-Surface.md and

services/workspace/src/ws.ts

@@ -19,6 +19,27 @@ import { dispatch } from './capabilities';
 import { dispatchChatTurn } from './chat';
 import { getNats } from './nats';
 
+// Invoke durability across reconnects (gh #41). Results for invokes whose
+// socket died before delivery are stashed here, keyed by invoke id, and
+// handed over when the reconnected client sends a `claim` frame. In-flight
+// dispatches are tracked so a claim can attach to work still running.
+// Process-local by design — a workspace restart loses both maps, and the
+// claim then fails fast with an explicit retry message instead of hanging.
+const inflightInvokes = new Map<string, Promise<string>>();
+const completedInvokes = new Map<string, { frame: string; expires: number }>();
+const INVOKE_RESULT_TTL_MS = 15 * 60_000;
+
+function stashResult(id: string, frame: string): void {
+  completedInvokes.set(id, { frame, expires: Date.now() + INVOKE_RESULT_TTL_MS });
+}
+
+setInterval(() => {
+  const now = Date.now();
+  for (const [id, entry] of completedInvokes) {
+    if (entry.expires < now) completedInvokes.delete(id);
+  }
+}, 60_000).unref();
+
 const BROADCAST_SUBJECTS = [
   'record_set.created',
   'record_set.deleted',
@@ -168,21 +189,76 @@ export async function registerWebsocket(app: FastifyInstance): Promise<void> {
       };
 
       // --- invoke frame: existing capability dispatch path. ---
+      // Long dispatches (didi crawls run minutes) must survive the caller's
+      // socket dropping mid-flight: the serialized result frame is tracked
+      // in-flight and, if the socket is gone when it lands, stashed for a
+      // post-reconnect `claim` frame (gh #41). A workspace restart still
+      // loses both maps — the claim then fails fast and explicit instead of
+      // hanging the caller forever.
       if (f.kind === 'invoke' && f.id && f.capability) {
-        try {
-          // Actor attribution envelope (build-order step 4) — the verified
-          // didi.sh identity rides beside the args into dispatch(), never
-          // client-asserted. See [[Workspaces-as-Tenant-Primitive]] §
-          // "Tenant-aware envelope" for the sibling client_id pattern.
-          const actor = session.didi
-            ? { didi_id: session.didi.didi_id, ...(f.via ? { via: f.via } : {}) }
-            : undefined;
-          const result = await dispatch(f.capability, f.args ?? {}, actor);
-          socket.send(JSON.stringify({ kind: 'result', id: f.id, ok: true, result }));
-        } catch (err: unknown) {
-          const error = err instanceof Error ? err.message : String(err);
-          socket.send(JSON.stringify({ kind: 'result', id: f.id, ok: false, error }));
+        const invokeId = f.id;
+        // Actor attribution envelope (build-order step 4) — the verified
+        // didi.sh identity rides beside the args into dispatch(), never
+        // client-asserted. See [[Workspaces-as-Tenant-Primitive]] §
+        // "Tenant-aware envelope" for the sibling client_id pattern.
+        const actor = session.didi
+          ? { didi_id: session.didi.didi_id, ...(f.via ? { via: f.via } : {}) }
+          : undefined;
+        const resultPromise = (async () => {
+          try {
+            const result = await dispatch(f.capability as string, f.args ?? {}, actor);
+            return JSON.stringify({ kind: 'result', id: invokeId, ok: true, result });
+          } catch (err: unknown) {
+            const error = err instanceof Error ? err.message : String(err);
+            return JSON.stringify({ kind: 'result', id: invokeId, ok: false, error });
+          }
+        })();
+        inflightInvokes.set(invokeId, resultPromise);
+        const resultFrame = await resultPromise;
+        inflightInvokes.delete(invokeId);
+        if (socket.readyState === 1 /* OPEN */) {
+          try {
+            socket.send(resultFrame);
+          } catch {
+            stashResult(invokeId, resultFrame);
+          }
+        } else {
+          stashResult(invokeId, resultFrame);
+        }
+        return;
+      }
+
+      // --- claim frame: re-attach to an invoke after a reconnect. ---
+      if (f.kind === 'claim' && f.id) {
+        const claimId = f.id;
+        const done = completedInvokes.get(claimId);
+        if (done) {
+          completedInvokes.delete(claimId);
+          socket.send(done.frame);
+          return;
+        }
+        const inflight = inflightInvokes.get(claimId);
+        if (inflight) {
+          const resultFrame = await inflight;
+          // The original waiter may have stashed it between our lookup and
+          // resolution — drop any duplicate stash and deliver here.
+          completedInvokes.delete(claimId);
+          try {
+            socket.send(resultFrame);
+          } catch {
+            stashResult(claimId, resultFrame);
+          }
+          return;
         }
+        socket.send(
+          JSON.stringify({
+            kind: 'result',
+            id: claimId,
+            ok: false,
+            error:
+              'invoke not found — the workspace service restarted while it was in flight; retry the action',
+          }),
+        );
         return;
       }