Roll the hardened NATS consumer across all services — 112 loops still carry the idiom that killed domain.list

[mpstaton]

Follow-up to #83, which fixed the observed failure in record-surrealdb-resolver/src/domains.ts and left the pattern in place everywhere else.

Root cause, evidence, and the three defects: context-v/issues/One-Stuck-Message-Kills-A-NATS-Subject-Until-Restart.md

Scope

grep -c "for await (const msg of sub)" across services/*/src:

Service	Bare loops
record-surrealdb-resolver	39
row-store	22
content-ingest	14
response-store	10
prompt-runner	8
prompt-store	8
social-search	7
workspace	2
ingest	1
xlsx-ingest	1
Total	112

Correcting the record on #83

#83 says "only this service changed", which understates what is left inside that service. Only domains.ts was touched. Per file:

File	Bare loops	On serveSubject
domains.ts	7	7
handlers.ts	17	0
person-handlers.ts	13	0
org-relations.ts	1	0

So organization.*, affiliation.* and person.* — the subjects behind Org Workbench, the affiliation rating resolver, and person enrichment — all still carry the full defect. Any one of them can lose its subject for a process lifetime to a single malformed payload.

What is already reusable

services/record-surrealdb-resolver/src/nats-loop.ts has no dependency on that service. It takes any AsyncIterable of reply-shaped messages, which is what made the failure modes testable without a broker or a database. Exports serveSubject() and withDeadline(), with 9 tests in test/nats-loop.test.ts.

Lifting it means deciding where it lives. Options, in rough order of preference:

1. A shared workspace package (packages/nats-service or similar) that every service depends on. Cleanest, but services build dir-context with plain npm install and cannot see packages/ — see Sharing-Code-Without-Breaking-Microfrontend-Autonomy. That Dockerfile change is a prerequisite, not a detail.
2. Copy the file per service. Ugly, but honest given (1), and ~120 lines with no dependencies.
3. Leave it where it is and only harden the services that matter most.

Not a mechanical sweep

Each loop responds with its own payload shape, and some (domain.create, the fetch handlers) need a wider deadline because they make cross-service NATS requests with their own timeouts. Converting blind will break reply contracts. This wants doing service by service, with that service's suite green after each.

Suggested order

record-surrealdb-resolver finishes first (31 loops left, and it is the one that already failed in production), then row-store and content-ingest by volume, then the rest.

Also worth folding in

getDb()-style unbounded connect() currently exists only in record-surrealdb-resolver/src/surreal.ts and is already bounded. Any service that later adds a persistent DB connection should take the same deadline + shared-in-flight-handshake shape rather than reinventing it.