fix(crawl, workspace): the eternal spinner — crawl invokes get a client deadline, dispatch ceiling doubles

The Curry Foundation crawl completed server-side in 87s while the tab
spun forever: workspace.invoke has no client-side timeout, WS reconnects
(e.g. a workspace-service rebuild mid-crawl — three today) drop pending
invokes, and the 300s dispatch ceiling sat one slow team crawl away from
biting (211s observed live on Quell).

crawlSearch and crawlTeam now race a 660s deadline — the spinner always
resolves to a retryable error that names the run may still have finished
server-side — and organization.crawl's NATS timeout moves to 600s
(pack.fan_out's precedent). Transport-level invoke re-correlation across
reconnects folds into the liveness sweep (#21); crawl progress frames
(#35) would surface a dead reply path within seconds.

Closes #40.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvYzx7vDWeafnkAi2nEQeb

Author: mpstaton

apps/org-workbench/src/lib/org-client.ts

@@ -47,6 +47,21 @@ export async function saveBrief(client: string, brief: string): Promise<void> {
   if (!r.ok) throw new Error(r.error || 'client.brief.set failed');
 }
 
+// The workspace invoke has NO client-side timeout, and a WS reconnect drops
+// pending invokes — a lost reply means an eternal spinner. Long-running
+// calls race a deadline so the UI always resolves to retryable state.
+function withDeadline<T>(p: Promise<T>, ms: number, label: string): Promise<T> {
+  return Promise.race([
+    p,
+    new Promise<T>((_, reject) =>
+      setTimeout(
+        () => reject(new Error(`${label} got no reply in ${Math.round(ms / 1000)}s — the run may still finish server-side; re-fire to retry`)),
+        ms,
+      ),
+    ),
+  ]);
+}
+
 // didi's team crawl — candidates only; staging + accept live in PeopleReveal.
 export type CrawledPerson = {
   name: string;
@@ -60,11 +75,11 @@ export async function crawlTeam(
   org_slug: string,
   client: string,
 ): Promise<{ people: CrawledPerson[]; filtered_note: string; source_urls: string[] }> {
-  const r = (await workspace.invoke('organization.crawl', {
-    org_slug,
-    target: 'team',
-    client,
-  })) as {
+  const r = (await withDeadline(
+    workspace.invoke('organization.crawl', { org_slug, target: 'team', client }),
+    660_000,
+    'didi team crawl',
+  )) as {
     ok: boolean;
     people?: CrawledPerson[];
     filtered_note?: string;

apps/search-and-add/src/lib/search-client.ts

@@ -52,6 +52,21 @@ export async function scanStream(args: {
   };
 }
 
+// The workspace invoke has NO client-side timeout, and a WS reconnect drops
+// pending invokes — a lost reply means an eternal spinner. Long-running
+// calls race a deadline so the UI always resolves to retryable state.
+function withDeadline<T>(p: Promise<T>, ms: number, label: string): Promise<T> {
+  return Promise.race([
+    p,
+    new Promise<T>((_, reject) =>
+      setTimeout(
+        () => reject(new Error(`${label} got no reply in ${Math.round(ms / 1000)}s — the run may still finish server-side; re-fire to retry`)),
+        ms,
+      ),
+    ),
+  ]);
+}
+
 // v1.2 crawl mode — didi's web crawl for the launching org. One model turn
 // with server-side web search; slow (tens of seconds); candidates only.
 export async function crawlSearch(args: {
@@ -60,7 +75,11 @@ export async function crawlSearch(args: {
   client: string;
   max_results?: number;
 }): Promise<{ provider: string; results: ConnectorResult[] }> {
-  const r = (await workspace.invoke('organization.crawl', args)) as {
+  const r = (await withDeadline(
+    workspace.invoke('organization.crawl', args),
+    660_000,
+    'didi crawl',
+  )) as {
     ok: boolean;
     provider?: string;
     results?: ConnectorResult[];

context-v/issues/Crawl-Replies-Can-Be-Lost-Eternal-Spinner-No-Client-Timeout.md

@@ -0,0 +1,57 @@
+---
+title: "Crawl replies can be lost — the eternal 'crawling…' spinner (no client timeout, reconnect-dropped invokes, tight dispatch ceiling)"
+lede: "The Curry Foundation crawl finished server-side in 87 seconds — and the tab spun forever anyway. Three gaps stacked: the browser invoke has no deadline, a WS reconnect (e.g. a workspace-service rebuild mid-crawl) silently drops pending invokes, and the 300s dispatch ceiling was one slow team-crawl away from biting (211s observed)."
+date_created: 2026-07-24
+date_modified: 2026-07-24
+authors:
+  - Michael Staton
+augmented_with:
+  - Claude Code on Claude Fable 5
+semantic_version: 0.0.0.1
+tags:
+  - Issue
+  - Augment-It
+  - Didi-Crawl
+  - Workspace
+  - Transport
+  - Reliability
+status: Shipped
+date_first_published: 2026-07-24
+post_ship_note: "Mitigations shipped 2026-07-24 — 660s client-side deadline on both crawl invokes (spinner always resolves to a retryable error naming that the run may still finish server-side), dispatch ceiling 300s→600s. The root transport gap (re-correlating pending invokes across WS reconnects) is folded into the legibility/liveness sweep scope ([[Live-Not-Live-Indicator-Tooling-And-Cross-Service-Error-Surfacing]]); [[Crawl-Progress-Is-A-Black-Box-Needs-Traces-The-Operator-Can-Watch]]'s progress frames would also make a lost reply visible within seconds. gh #40 closed."
+---
+
+# Crawl replies can be lost
+
+## The incident (2026-07-24 evening, logs-confirmed)
+
+Operator fires a links crawl on The Beth & Ravenel Curry Foundation → UI
+stuck on "crawling…". prompt-runner logs show the truth: `crawl completed
+… ms: 87542`. The work succeeded; the reply never made it back. Earlier
+team crawls (Quell 211s, Truist 147s) completed too — 211s uncomfortably
+close to the 300s dispatch ceiling.
+
+## The three stacked gaps
+
+1. **No client-side deadline.** `workspace.invoke` waits forever; a lost
+   reply is an eternal spinner with a disabled button.
+2. **WS reconnects drop pending invokes.** The transport reconnects with
+   backoff, but in-flight invokes don't survive the new socket — and
+   workspace-service container rebuilds (three today) sever every session.
+   Any crawl pending across one is orphaned.
+3. **300s dispatch ceiling vs multi-minute crawls.** The NATS
+   request timeout was set before live team-crawl timings existed.
+
+## Shipped mitigations
+
+- 660s `withDeadline` race on `crawlSearch` and `crawlTeam` — the spinner
+  always resolves; the error names that the run may still have finished
+  server-side and to re-fire.
+- `organization.crawl` dispatch ceiling 300s → 600s (pack.fan_out's
+  precedent).
+
+## Remaining (folded elsewhere)
+
+- Transport-level invoke re-correlation across reconnects (or
+  server-held results claimable by request id) → the liveness sweep (#21).
+- Crawl progress frames (#35) would surface a dead reply path within
+  seconds instead of minutes.

services/workspace/src/capabilities.ts

@@ -306,8 +306,9 @@ const CAPABILITY_TIMEOUTS_MS: Record<string, number> = {
   'client.brief.get': 30_000,
   'client.brief.set': 30_000,
   // A crawl is one model turn with multiple server-side web searches (plus
-  // pause_turn continuations) — minutes, not seconds.
-  'organization.crawl': 300_000,
+  // pause_turn continuations) — minutes, not seconds. Team crawls have been
+  // observed at 211s live; 600s matches the pack.fan_out ceiling.
+  'organization.crawl': 600_000,
   // One query, one provider — pack.search's budget.
   'search.fire': 30_000,
   // Multi-stage (Firecrawl index harvest + per-post dates + dedup read) —