Skip to content

feat(102): schema-deferred direct mode — complete (89/89 tasks) - #1063

Merged
Dumbris merged 62 commits into
mainfrom
feat/102-schema-deferred-impl
Aug 28, 2026
Merged

Dumbris merged 62 commits into
mainfrom
feat/102-schema-deferred-impl

Conversation

@Dumbris

@Dumbris Dumbris commented Aug 26, 2026 •

Copy link
Copy Markdown
Member

Implements T004 + Phases 1–3 of specs/102-schema-deferred/tasks.md. Draft — Phases 1–3 of 7. Related #971.

Based on 102-schema-deferred-plan (#1045) so the spec artifacts travel with the work; retarget to main once that merges.


Phase 7 — polish, and the numbers

Spec 102 is complete: 89/89 tasks.

The direct built-ins gate (T073/T074)

Standalone, deliberately not an entry in toolsListGoldenSurfaces — that list is driven by _DeltaIsEnumerated, which reads a frozen pre099/<surface>.json per surface, and direct mode has no pre-feature baseline because nothing was registered on it before this spec. With zero upstreams the listing is exactly describe_tool, pinned byte-exact in both modes together with the instructions string the surface now carries.

T074's point is real: directServer is the one routing-mode server carrying WithToolFilters, so registered and served are different questions here. The gate captures both and fails on any difference.

SC-001, measured two ways, and they agree

T075 added the first bench arm that renders a tools/list payload rather than a retrieve_tools result set — which is why the profiler could not price this feature before. Run through the profiler itself:

arm tokens savings vs baseline
baseline_json 4386 —
compact_sig 2081 52.6%
direct_deferred 3090 29.5%

The in-process gate, which measures the real renderDirectTools output rather than an arm's re-encoding, says 29.7%. These are not the same measurement — the arm emits canonical JSON without annotations or mcp-go's _meta, the gate counts the real marshalled mcp.Tool including both (4386 vs 6116 full-mode tokens for the same 45 tools). What matters is that two independently-built paths land within 0.2 percentage points of each other, and both are less than half of what SC-001 asks for. SC-002 is 86.7% against its 80% floor.

So SC-001's ≥70% is confirmed unreachable independently of the implementation, by the spec's own sanctioned pipeline. The ≥70% assertion T076 asks for is deliberately not added: a test that cannot pass is not a gate. This is the one item left that needs a maintainer decision — see the escalation under Phase 3 for why the ceiling is arithmetic (38.9% even if the schema and the signature were both deleted).

E2E (T077)

Three tests driving a real client over HTTP through the real POST /api/v1/config/apply path:

  • SC-006 — a client that is already initialized and has already listed the tool set sees the next tools/list reflect the flip. It is deliberately never reconnected: a reconnect would prove only that a fresh session is built from current config, which is the easy half.
  • SC-003 — a wrong guess returns invalid_params with the full schema embedded; the retry corrected from that schema reaches dispatch.
  • FR-003 — the legacy aliases are asserted against /mcp/all rather than a hand-written expectation, so the test states the invariant instead of re-encoding the renderer.

Mutation-checked: disabling the reload guard fails both the live-flip and the alias tests.

T083 — verified, not assumed

All five frozen gates pass. Golden churn across the entire branch is exactly the two enumerated regens plus four new files; code_execution_mode.json and both pre099/ baselines are untouched, which is what confines the change to the two surfaces that register describe_tool.

go test -race ./internal/... is green except TestResolveDockerStatusResolvableAndWorking, which passes in isolation and on a re-run of its own package — it writes a fake docker into a TempDir and manipulates PATH, so it is PATH-sensitive under a parallel whole-tree race run. It never touches the direct surface.

Docs

docs/configuration.md, a new docs/features/schema-deferred-direct-mode.md (wired into website/sidebars.js), and two CLAUDE.md edits. The feature doc's client-compatibility section is the part worth reading before enabling this: schema-driven form UIs render an empty form under deferral and should stay on "full".


Phase 6 (US4) — operator opt-in, hot reload, and the skew proof

direct_tool_response_mode is now flippable on a running proxy. Verified live in one process, by editing the config file:

BEFORE  desc: "[gh] Create a new issue in a repository."
        schema: {"properties":{"labels":…,"owner":…,"title":…},"required":[…]}

>>> flip to deferred — no restart

AFTER   desc: "[gh] Create a new issue in a repository.\ncreate_issue(owner*:str, repo*:str, title*:str, labels:[str], milestone~:obj)"
        schema: {"type":"object"}

…catalog generation 2 → 3. An unrelated edit (tool_response_limit) in the same session rebuilt nothing.

Why the setting needed more than a detector clause

Unlike tool_response_mode, the direct listing is registered state — reading the live config on the next call changes nothing, so the surface has to be rebuilt. That makes the guard load-bearing: SetTools re-registers everything and emits notifications/tools/list_changed to every initialized session, so an unguarded rebuild would make any config edit look, to a client, exactly like the tool set changing.

The guard compares against the catalog, which now records the mode it was rendered with. Only the snapshot knows what connected clients were actually served; config only says what the operator wants. The failure paths stamp it too, so a flip made while discovery is down is not lost.

T070 corrected a claim in D13

The skew tests stage the real window — render, SetTools, pause, publish — and measure what a concurrent session sees. They found that "the filters deny it, which is safe" holds for scoped sessions only: filterDirectModeToolsForAuth short-circuits for a session with no agent token and no active profile, so an unscoped one is served the raw registry and never consults the catalog.

case scoped session unscoped session
name added denied on both sides listed, but describe says not_found
name removed still describable from the previous snapshot until the publish same

Neither leaks — the first is a session already entitled to the whole surface, the second is a definition that session was served a moment earlier — and both close at the publish. The comment in RefreshDirectModeTools now says what the window does rather than what it was hoped to do, and tasks.md records it beside the checkpoint.

The three accepted residuals (T002/T003) are reproduced as specified, including the signature-identical nested-schema edit — a canonicalization-equal edit will not do, since hash re-marshals the parsed schema. The annotations-only case proves its compensating property: a read-scoped token still sees the stale listing, but its call is refused, because dispatch re-derives the tier from the annotations its own registration captured and never reads the catalog.

Cross-model review

opencode (gpt-5.6-sol) found five real defects, fixed in 6cbba5772 — and three of them were tests proving less than they claimed:

Finding Why it mattered
Check-then-lock race in the reload guard Two concurrent reloads could both see drift and publish two generations — the churn the guard exists to prevent, reintroduced by its own racy read
"" vs "full" reported as a changed field Deleting the key claimed a change that did not happen; the guard suppressed the churn but the apply result still misreported it
The origin-flip test was vacuous It flipped alpha__run → beta__run — different display names, so not an origin flip at all, and its loop dispatched nothing
Both schema-residual tests ran in full mode Where a schema change is plainly visible, so "silently stale" was shown by a listing that was not silent
The annotations residual built its own handler Proving only that a test-constructed handler refuses, not the registered one

The origin-flip test now uses the only construction that expresses one — the __ ambiguity, where a__b__c is either (a, b__c) or (a__b, c) — and asserts what actually matters: during the window the stale catalog scope-checks against the old origin and still lists the name, but the registered handler refuses the call naming the new one. The schema residuals run deferred and additionally assert the wire bytes are byte-identical across the change, which is what makes the staleness silent. Every fix was mutation-checked by reverting it.

Its sixth finding is documented rather than changed: a flip made while discovery is failing publishes an empty catalog stamped with the new mode — which T069 requires — so the guard will not retry. Deliberate: the empty listing is itself the visible problem, and the servers.changed that fixes discovery rebuilds unconditionally.

Separately, the SetTools-then-publish order is now pinned by a test rather than by a comment. It was previously untestable: once a rebuild completes both land consistently whichever order they went in, so inverting them failed nothing. A three-line seam (nil in production) lets a test observe the real publisher mid-flight; inverting the order now fails.

Verification

internal/{server,runtime} green under -race with the CI skip regex · golangci-lint v2 clean repo-wide · -tags server build OK · live hot-reload and no-churn both confirmed on an isolated instance.


Phase 5 (US3) — self-healing direct calls

A deferred listing advertises {"type":"object"}, so an agent guesses its arguments from a compact signature. This phase decides what being wrong costs: an opaque upstream error is an unbounded debugging loop; a pre-dispatch rejection carrying the full schema is exactly one retry.

Verified live, end to end:

// 1. the agent guesses wrong — omits the required "title"
{ "error_type": "invalid_params",
  "error": "invalid arguments for gh__create_issue: at '': missing property 'title'",
  "hint":  "Fix arguments to match input_schema, then retry. For the full definition call describe_tool({tool_ids:[\"gh__create_issue\"]}).",
  "input_schema": { "properties": {…}, "required": ["owner","repo","title"], "type": "object" },
  "tool": "gh__create_issue" }

// 2. the corrected retry
{ "tool": "create_issue", "ok": true }

…and both appear in the activity funnel — the rejection as error carrying its reason, the retry as success.

Three decisions worth naming

  • The stored schema, never the placeholder. Validating against what the listing advertised would accept everything and validate nothing. It is also mode-independent: a full-mode client that guessed wrong gets the same help.
  • The caller's args, not enrichedArgs. injectAuthMetadata adds properties the upstream schema never declared; a schema with additionalProperties:false would reject them, turning the proxy's own bookkeeping into the agent's bug on a call whose arguments were correct.
  • Validation and its activity emission land together, as T064 insists. Landing the first alone would start rejecting calls that appear nowhere in the funnel — the blind spot Improve retrieve_tools diagnostics for annotation filters and automation-critical tools #969 established this handler must not have. markSessionWorked stays after it: a rejected call did no upstream work, so it earns no durable work session, like the policy blocks above it.

T019a was ticked but never done

makeDirectModeHandler still took (serverName, toolName, annotations) when Phase 5 arrived looking for the captured entry — the exact failure that task was written to prevent. It is landed here: the handler closes over its own *directCatalogEntry, so a request in flight keeps the definition it was dispatched under even as the catalog is swapped underneath it (R9). T020's tick had the same problem; its four call sites are updated now.

Cross-model review

opencode (gpt-5.6-sol) found one real defect: the "in both serialization modes" subtest set the mode and then built the handler by hand — but the mode is read inside renderDirectTools and nowhere else, so both subtests ran identical code and the cross-mode claim was vacuous. Each subtest now dispatches through the handler the real render path produced, and asserts the two modes advertised different schemas on the wire. Its other seven answers found no defect.

Two more vacuous tests were caught before the review, by mutation-testing: the additionalProperties guard passed for the wrong reason twice (no auth context to enrich with, then a token whose empty allowed_servers blocked the call before it reached the validator).

Verification

internal/{server,runtime} green under -race with the CI skip regex · golangci-lint v2 clean repo-wide · -tags server build OK · removing the validation fails five tests; making the renderer ignore the mode fails the cross-mode test.


Phase 4 (US2) — describe_tool on the direct surface

Catalog → Inspect → Execute is complete on one surface: a deferred listing hands an agent a compact signature, and describe_tool hands back the full schema for the same id, in the same grammar.

// POST /mcp/all — describe_tool {"tool_ids": ["gh__create_issue"]}
{ "definitions": [ { "name": "gh:create_issue", "server": "gh",
                     "call_with": "call_tool_write",
                     "annotations": { "destructiveHint": false, "openWorldHint": true, … },
                     "inputSchema": { "properties": { "owner": …, "repo": …, "title": … },
                                      "required": ["owner","repo","title"], "type": "object" } } ],
  "errors": [] }

(Captured from a live isolated instance.)

The resolver is catalog-backed, and that is the whole point

Every other surface resolves describe_tool ids through the search index. That is the wrong authority here in both directions:

  • The index is filtered at the tool level, while the direct listing is filtered at the server level and deliberately retains pending/changed tools for an operator. An index-backed resolver therefore answers not_found for ids the same session can see in its own tools/list — making deferral strictly worse than full mode, since the agent could then neither read the schema from the listing nor recover it from describe.
  • Conversely the index holds tools from servers the direct listing never projected, which would be a disclosure.

So the direct surface resolves through the catalog — the exact snapshot tools/list rendered from. The surface is bound at registration, not sniffed from the request (which would be spoofable); one builder and one response shape still serve both.

Listing parity is structural, not asserted

The scope+tier gate and the callability gate are now shared with the tools/list filters rather than mirrored. T057 probes every catalog entry in both grammars across four session shapes (admin, read-scoped, write-scoped, profile-pinned), in both directions: no id may be describable-but-unlisted (a disclosure) or listed-but-undescribable (SC-007). Disabling the tier gate fails it.

Neither id form is derived from the other — re-parsing we__ird__do_thing on the first __ names a server that does not exist. Swapping the resolver for a re-parsing one fails the suite.

Check mode

Ids are canonicalized in and the caller's own ids and ordering restored out, in the response and the activity record (D14). Invisible ids are gated before the evaluator — it knows nothing about the permission tier or a withheld collision and would answer ready or server_quarantined for an id the listing says does not exist — and answer through preflight.NotFound, so a gated id is byte-indistinguishable from an absent one. Visible ids project the evaluator's status/reason/action/retryable through untouched.

Prose, budget and the frozen gates

Five strings named retrieve_tools, which does not exist on this surface — they were telling a stuck agent to call a tool it cannot see. All five are surface-neutral and tool_ids names both accepted forms. That overran the 250-token budget (252); prose was shortened rather than the ceiling raised (248 now), per the rule the 150→250 bump set.

Exactly the two enumerated goldens were regenerated; code_execution_mode.json diffs empty and both pre099/ baselines are untouched. describePlainDelta widened from one substitution per scenario to an ordered slice — an out-of-scope id now carries two independent changes and collapsing them would let one pass unenumerated.

Cross-model review

opencode (gpt-5.6-sol) reviewed Phase 4. Three findings were real defects, fixed in bcfb8cf92, each with a test that fails without it:

Finding Why it mattered
Two caller ids naming one tool gh__read_file + gh:read_file in one batch: the second was gated not_found — "gone" for a tool the caller had just been listed
Display/canonical namespace overlap x__y:z is server x+tool y:z and server x__y+tool z; the canonical id silently answered with the wrong tool's definition
: in a server name preflight ids split on the first colon, so canonicalizing made the evaluator answer confidently about another tool

Also tightened while answering its questions: one catalog snapshot pinned per call (a rebuild could answer two ids of a batch from different generations), a resolved slice in the evaluator's index reader instead of a storage-backed predicate re-run per entry, and no reverse map scan in restore.

Its remaining findings resolve to documented, already-accepted properties — output_schema on the indexed surface is the contract (D2) and gate 4 passes unregenerated; did_you_mean formatting differs between gated and evaluated not-founds but both draw from the same session-visible snapshot; and the SetTools-then-publish parity window is D13's one-directional guarantee accepted at T001, failing in the safe direction. All three are written into tasks.md beside the Phase 4 checkpoint.

Verification

internal/{server,preflight,runtime} green under -race with the CI skip regex · golangci-lint v2 clean repo-wide · -tags server build OK · live isolated instance: a direct id returns the full schema, check mode answers both grammars under the caller's own ids with catalog-drawn did_you_mean, definition mode corrects case, and /mcp still rejects direct ids.


Phase 3 (US1) — deferred enumeration

With direct_tool_response_mode: "deferred", every upstream tool is still listed — same names, count, ordering source and annotations — but each entry carries its Spec-085 compact signature and a minimal permissive schema instead of the upstream inputSchema/outputSchema.

{
  "name": "gh__create_issue",
  "description": "[gh] Create a new issue in a repository.\ncreate_issue(owner*:str, repo*:str, title*:str, labels:[str], milestone~:obj)",
  "inputSchema": { "type": "object" },
  "annotations": { "readOnlyHint": false, "destructiveHint": false, "idempotentHint": false, "openWorldHint": true }
}

(Captured from a live isolated instance, not hand-written.)

Why mcp.NewToolWithRawSchema

mcp.NewTool cannot produce the FR-004 wire shape: its marshaller always emits "properties":{} and "required":[], which re-opens the arg-pruning hazard the placeholder exists to close. So deferred entries go through the raw-schema constructor — which accepts no ToolOptions and seeds no annotation defaults, where NewTool seeds readOnly=false, destructive=true, idempotent=false, openWorld=true before any option runs. directToolAnnotations reproduces those defaults and then applies the upstream overrides; without it the two modes marshal a different annotations object for the same tool (D9). Asserted byte-for-byte across four fixtures, including a Title-only one.

The full-mode path moves out of the render loop verbatim (FR-015).

Signatures come from Peek, never Get

Get compiles and memoizes on a miss — that would put per-request compilation on the listing path FR-005 exists to keep it off, and would hide the miss entirely. On a miss the whole suffix is absent and the entry is otherwise unchanged: never dropped, never delayed.

Instructions on the direct server (FR-007)

No routing-mode server carried instructions before; the direct server now does. The operator's configured value when set, else a direct-specific default that names only server__tool calling and describe_tool — never retrieve_tools/call_tool_*/upstream_servers, none of which this surface registers (D16) — then the deferral legend, in both modes. Instructions are construction-time (mcp-go fixes WithInstructions on the instance), matching the default server.


⚠️ A bug only the live run found — and the fix

Unit tests were green on a build whose deferred listing carried no signatures at all.

warmSignatureCache hung off the add/modify branches of applyDifferentialToolUpdate. On any restart against an existing index both branches are empty — the differential update finds nothing to do — so the cache stayed empty for the life of the process. Spec 085 never noticed: its compact retrieve_tools reads through the compiling accessor and merely paid a first-call compile. Spec 102 reads through Peek, which never compiles, so every entry silently lost its signature while the listing still looked well-formed.

Fixed by warming a server's whole allowed set on every discovery (idempotent; Warm returns on the first hit). renderDirectTools now also logs signature_misses per rebuild, because a missing suffix is otherwise indistinguishable in the payload from a tool with no parameters.

Regression test: TestApplyDifferentialToolUpdate_WarmsUnchangedToolsOnRestart. Filed as T035b — found by live verification, not by any planned task.


⚠️ SC-001 is not reachable — escalating as a spec-numbers defect (T076)

T035a ran the gates. SC-002 passes: 39/45 = 86.7% one-shot callable, floor 80%. SC-001 does not, and cannot on either dataset in this repo. Measured with the real renderer and the profiler's pinned cl100k_base encoder:

corpus tools full deferred reduction SC-001 asks
corpus_v2.tools.json (frozen reference) 45 6116 4301 29.7% ≥70%
livemcptool_snapshot/tools.json 527 99918 65138 34.8% ≥85%

The shortfall is arithmetic, not an implementation shortfall. On corpus_v2 the upstream inputSchema is 2604 of 6116 tokens (42.6%); the rest is description (1784), mcp-go's unconditional annotations block (1125, ~25/entry, untouchable by any serialization change), names (286) and punctuation. Deferral removes the schema and adds the FR-004 signature, so even deleting the schema and the signature outright caps at 38.9% — below SC-001's floor. FR-004 forbids closing the gap by truncating descriptions, and Spec 085's first-sentence grammar only reaches ~68% here.

SC-001's numbers assume the schema is a far larger share of a tools/list payload than it is in either dataset. Resolution needs a maintainer decision at T076 — restate the threshold per corpus shape, or re-target the criterion at a schema-heavy corpus. Meanwhile TestDeferredDirect_TokenReduction_Corpus45 asserts a 25% regression floor plus an upper guard that fails if a future change ever does clear 70%, so the note cannot go stale silently.


FR-015 byte-stability, proved against a real pre-feature binary

TestDirectFullMode_ByteStableAgainstPreFeatureE2E drives the Spec-098 stdio fixture over raw JSON-RPC and compares against testdata/direct_full_prefeature.golden.json. That golden was produced by running the same test file, unmodified, against a merge-base binary — the file compiles and runs at 4d306240 (verified), so the baseline is a genuine pre-feature capture rather than the branch restating itself. Falsified by tampering with the golden before trusting it.


Cross-model review

opencode (gpt-5.6-sol) reviewed the Phase 3 commit. Three findings fixed in 8e1bf26c0: the E2E helper mis-decoded streams carrying a notifications/tools/list_changed frame alongside the reply; the map-keyed comparison laundered array order (writing that assertion corrected a wrong assumption — mcp-go sorts the whole served array by name, built-ins included); and cross-mode annotation parity used JSONEq where FR-004 wants bytes. Its "instructions are not hot-reloadable" finding is a documented limitation, not a regression — the direct server carried no instructions at all before.

Verification

  • go test ./internal/{server,runtime,toolsig,config} -race with the CI skip regex — green
  • golangci-lint run --config .github/.golangci.yml ./... — 0 issues
  • go build -tags server — OK
  • Live isolated instance (high port, scratch data-dir and config, never touches ~/.mcpproxy): both modes confirmed, FR-008 set identity and byte-identical annotations across the flip, signature_misses: 0 on a warm start

Based on 102-schema-deferred-plan (#1045) so the spec artifacts travel with the work; retarget to main once that merges.

No behaviour change yet

The axis exists and validates, and the signature cache gained a read path. Nothing reads either on a serving path — that starts in Phase 2.

T004 — spec.md had unresolved markers its own plan treated as decided

Three [NEEDS CLARIFICATION] blocks closed, each recording the rejected alternative:

  • D1 — a dedicated direct_tool_response_mode key, not an extension of tool_response_mode.
  • D2 — deferred entries strip outputSchema; describe_tool gains an additive output_schema.
  • D3 — the 5-id batch cap stays (50 with check:true), identical on every surface.

T005–T010 — the config axis

A separate axis on purpose. Reusing tool_response_mode would silently change /mcp/all output for every deployment already running compact, which FR-015 forbids. The existing flag and env alias keep their current meaning and their current help text — that non-change is itself a requirement.

TestDirectToolResponseModeEnvOverride/the_two_axes_are_independent asserts neither alias can move the other. Without it the "dedicated axis" decision is only skin deep.

FR-002: routing_mode: "schema_deferred" — the shape #971 proposed — now fails with a message naming the supported composition rather than listing three legal modes:

invalid routing mode: schema_deferred (must be retrieve_tools, direct, or code_execution);
"schema_deferred" is not a routing mode — use routing_mode: "direct" with
direct_tool_response_mode: "deferred"

A companion test checks that hint does not leak onto ordinary invalid values.

T011–T012 — toolsig.Cache.Peek

Not Get with different arguments. Get compiles and memoizes on a miss, which would:

  1. put per-request compilation on the tools/list path FR-005 exists to keep it off, and
  2. make the miss invisible — a caller that always receives a Signature cannot distinguish "warmed at index time" from "compiled just now for you", so it could never render the signature-less entry FR-004 requires.

The tests assert the two properties that make it different: CompileCount does not move, and the cache does not grow. Plus one that an evicted hash is not resurrected.

One task-list correction

T078 (make swagger) was scheduled in Polish. The pre-push hook runs swagger-verify and rejects the push the moment the config struct changes, so it lands here with T005. tasks.md updated to say so.

Verification

  • both editions build (go build ./..., -tags server)
  • go test -race across internal/config, internal/toolsig, cmd/mcpproxy
  • golangci-lint run --config .github/.golangci.yml → 0 issues

Next

Phase 2 (the directCatalog). T013–T016 are unblocked; T017–T020 wait on the T001 approval, since catalog publication is what that narrowing governs.

Dumbris added 30 commits August 25, 2026 06:58
Plan for schema-deferred direct-mode enumeration (Spec 102): dedicated
direct_tool_response_mode config axis, FR-017 directCatalog snapshot design,
describe_tool on the direct surface with listing-parity visibility,
pre-dispatch validation reuse, frozen-golden handling, and file-touch list.
Phase 0/1 artifacts: research.md, data-model.md, quickstart.md, contract.

Related #971
…findings

Self-review + cross-model review pass over the stage-2 plan artifacts.
Every item below was verified against this tree before recording.

Corrections to false/incomplete claims:
- R11 (new): the {"type":"object"} placeholder is NOT what mcp.NewTool
  produces. mcp-go v0.57.0's schema marshaller always writes properties:{}
  and required:[], which re-opens the arg-pruning hazard FR-004 exists to
  close. Deferred entries must use mcp.NewToolWithRawSchema with annotations
  copied on explicitly; mixing RawInputSchema onto a NewTool result is a hard
  marshal error, not a silent override. Verified empirically.
- R2: retracts "no golden pins describe_tool response bytes" — the
  describe_plain_corpus gate pins 18 response bodies byte-for-byte. Added as
  the FOURTH frozen gate with named describePlainDelta substitutions.
- R5: the surface-neutral prose rewrite covers FOUR retrieve_tools strings,
  not one — both parameter descriptions and the two shared remediation
  constants. tool_ids prose must also name the server__tool id form (FR-011).
- Test strategy: names TestDescribeTool_RegisteredInRetrieveToolsModeOnly's
  "direct routing mode" subtest, which asserts the v1 decision this feature
  reverses and will go red; and warns the new direct built-in gate must be
  standalone, since DeltaIsEnumerated demands a pre099 baseline per surface.
- toolsListAllowedDelta needs no edit: describe_tool is already enumerated
  there from spec 099.

Missed interactions now closed:
- D10: parity must hold on BOTH sides. The direct listing filters still
  resolve via ParseDirectToolName's first-__ split, so a server named a__b
  is judged as server a by the listing and as the real pair by a
  catalog-backed describe — describable-but-unlisted, the exact FR-011
  disclosure. Filters now resolve through the catalog. Also gated: the
  preflight did_you_mean corpus (server-level filtered only, no permission
  tier) and the definition-mode case-correction hint.
- D11: FR-007 instructions ignored the operator-configurable instructions
  config key, which today reaches only the default server. Direct-server
  instructions are now resolveInstructions(cfg) + the deferral legend.
- D12: upstream.Manager is concrete and lists only connected clients, so the
  unit matrix had no seam. buildDirectModeTools splits into pure
  buildDirectCatalog + renderDirectTools.
- Pre-dispatch validation placement pinned, including the started+completed
  activity pair the 085 path emits (issue #969 blind-spot property).
- FR-014 guard: publish an empty catalog on DiscoverTools failure so a flip
  during an upstream outage is not lost.

Related #971
…02 gates need

The token gates were specified as "measure via the spec-083 profiler pipeline"
with no implementation surface. bench/arms/ is a registry of encoding arms
(baseline, compact_sig, toon, tron, tscg) that all render retrieve_tools result
sets; none renders a direct tools/list, so the deferred measurement needs a new
registered arm plus its testdata render golden. Added bench/arms/ to the
file-touch list and the package count.

Related #971
… the FR-007 instructions precedence in the summary

Related #971
…return path; instructions hot-reload caveat stated

buildDirectModeTools returns nil early when DiscoverTools fails
(mcp_routing.go:81-85), so "append describe_tool inside tool-set construction"
must mean every return path or the built-in vanishes on the first upstream
hiccup. Also records that instructions are constructor-time on both server
instances, so D11 does not promise hot-reload it cannot deliver.

Related #971
…ture actually runnable

The merge-base capture as written cannot run: the D12 pure seam does not exist
at origin/main and DiscoverTools only returns tools from connected clients, so
there is no way to render a deterministic direct listing there from a fixture
slice. Replaced with a live stdio-fixture capture (the preflight_e2e_test.go /
testdata/preflight_fixture_server.js harness, which runs unchanged at the
merge-base) plus a same-tree differential fallback, with the trade-off stated.

Related #971
…tually is

The contract example showed create_issue(owner*:str, ...) while
toolsig.Signature.Sig is the parenthesised parameter list only and carries no
tool name (internal/toolsig/signature.go:18). Made the name prefix normative and
recorded that Signature.Desc is deliberately unused here, since deferred entries
keep the full description rather than the 085 first-sentence prefix.

Related #971
…rral legend

R4 and the contract carried two different draft legend texts. R4 now points at
the contract as the single source so the pinned bytes have one origin.

Related #971
Mirrors effectiveToolResponseMode (profile_resolver.go:57) so the renderer, the
catalog's recorded serializationMode, and the FR-014 rebuild guard cannot resolve
the mode by three different paths. Notes it takes no detail override, since
tools/list has no parameters.

Related #971
…nition annotations)

buildFullToolEntry does not read annotations from the entry passed in: it calls
p.lookupToolAnnotations (mcp.go:6344), which reads the StateView snapshot, and
derives call_with from that, defaulting to ToolVariantRead on nil. A
catalog-backed describe that synthesizes a ToolMetadata would therefore still
resolve annotations out-of-band — so the listed pending/changed destructive tool
SC-007 targets would describe with no annotations and call_with:"read". That is
a wrong safety hint, not just a missing field. The definition-assembly seam now
takes a catalog-supplied annotations override on the direct surface only, leaving
the retrieve_tools path byte-identical for the frozen goldens.

Related #971
…the direct check-mode corpus

preflight.Scope is a server-name set, so the evaluator structurally cannot
express the per-tool gate the direct listing applies — which is why did_you_mean
leaks names the listing hides. The fix is concrete rather than aspirational:
preflight.EvalContext.Index is an interface (preflight_glue.go:103 supplies
preflightIndexReader), so the direct surface passes a catalog-backed reader and
both id resolution and the suggestion corpus share one authority. Suppression
demoted to fallback. Added preflight_glue.go to the file-touch list.

Related #971
…tution row with D9-D12

D4/D5 rows and the Principle IV assessment still described the pre-hardening
scope. Also records that all four direct routes share one p.directServer
instance (GetMCPServerForMode), so a single SetTools covers the aliases.

Related #971
…ndings

Round 2 (opencode / gpt-5.6-sol). Every finding was re-verified against this
tree before the fix.

BLOCKER — catalog/registry publication skew (new R13, D13). The atomic.Pointer
makes the catalog swap atomic but NOT the catalog-plus-mcp-go-registry pair:
RefreshDirectModeTools publishes the catalog and then calls SetTools
(mcp_routing.go:666-673), and mcp-go owns its own registry read, so no lock of
ours can span both. Handler-to-schema atomicity does hold (closure capture); the
filters and describe resolver are what skew. Closed by rule rather than denied:
catalog published first, filters DENY on catalog miss (built-ins matched by an
explicit name set — the previous fall-through wording would have let a stale
entry skip the permission-tier gate during the window), describe additionally
requires directServer.GetTool(name) != nil, plus a generation counter and an
interleaving test.

MAJOR — check-mode ids (D14). preflight.Evaluate accepts colon ids only
(splitToolID, evaluator.go:500-510) and normalizeDescribeCheckIDs forwards ids
verbatim, so every server__tool id under check:true would answer not_found no
matter what the id gate does. The adapter now canonicalizes before Evaluate and
restores the caller's original id and ordering in both response and activity.

MINOR — server-level vs tool-level states. DiscoverTools drops disabled and
quarantined SERVERS before projection (upstream/manager.go:1128-1138), so the
"non-agent listings retain quarantined/disabled" claim was too broad: only
pending / changed / tool-level-disabled reach the listing.

MINOR — TestRetrieveToolsFullMode_GoldenByteIdentity pins TWO goldens
(retrieve_full_default and retrieve_full_stats, mcp_entry_builder_test.go:116).
Both are now listed as must-pass-unregenerated.

Related #971
…empty one in the deny-on-miss rule

Deny-on-miss would otherwise blank the listing before the first publication and
break every hand-constructed test proxy that calls the direct filters directly.
A nil pointer keeps today's ParseDirectToolName behavior; an empty published
generation is real and denies.

Related #971
…icity claim

R9 buys handler-to-schema atomicity only; a reader stopping there would carry
away the over-broad claim R13 corrects.

Related #971
…indings

BLOCKER — D13 did not cover a same-name update. GetTool(name) != nil is
satisfied by the old entry while the resolver uses the new catalog, so a schema
edit or a flipped collision winner could be described one generation ahead of
the registry. Generation stamping is not available (mcp.Tool's only free-form
field marshals to _meta, which FR-015 freezes; mcp-go snapshots its tool map
before our filters run), so the plan now promises a stated safety property
instead of atomicity: SetTools first then publish, permission tier derived from
the registered entry's OWN annotations (generation-correct by construction,
closing the read-to-destructive over-permit), and the residual same-name
staleness enumerated with tests for all four interleavings.

MAJOR — the direct server registers nothing at init (mcp_routing.go:651-653,
"built lazily... servers.changed will trigger a refresh"). Composing the
built-in into rebuild construction alone leaves FR-009 false until the first
refresh, worst with zero upstreams, and leaves the catalog nil so the first
unrelated reload churns every client. Added D15/R14: an initial direct rebuild
in initRoutingModeServers, and the built-in gate now asserts over
p.directServer.ListTools() rather than the builder's return value.

MAJOR — D11 imported resolveInstructions' defaultInstructions, whose DISCOVERY
line says "Use 'retrieve_tools'" and whose CALLING line names call_tool_*, none
of which exist on the direct surface. D16: operator text when set, otherwise a
direct-specific default.

MAJOR — the check-mode column used definition mode's vocabulary. Check mode
answers status:"unavailable" plus a preflight.Reason (tool_pending_approval,
tool_changed, tool_blocked_by_user, tool_denied_by_config, ...). The adapter now
projects those through untouched, with a byte-compatibility test.

Related #971
… chosen ordering

The table still marked catalog-first as chosen after the round-3 fix reversed it.
Rewritten with the same-name row added and the reasoning for SetTools-first made
explicit: removal becomes window-free, addition fails closed.

Related #971
…t publish

Today buildDirectModeTools calls setDirectToolPermissions mid-build, before
RefreshDirectModeTools reaches SetTools — the exact ordering R13 rule 1
reverses. Made the constraint explicit so the split in D12 does not carry the
old publication point along with it.

Related #971
…ocument

Round 4 (opencode). Both findings were my own round-3 fixes not carried through.

D13 rule 3 (tier from the registered entry's own annotations) had not reached
data-model.md, which still routed the listing filters at the catalog's
requiredPermission and still asserted "installed by one SetTools call
(atomicity)" — the claim R13 retracted. The catalog now backs the (server,tool)
split and the describe resolver only; the invariant is restated as R13's safety
property.

D16 had not reached plan.md, whose FR-007 summary and D11 row still mandated
resolveInstructions — the function D16 exists to avoid. Named the single helper
resolveDirectInstructions and dropped upstream_servers from the direct default:
management tools are registered by the code-exec and call-tool builders only
(mcp_routing.go:398, :507), so naming it would repeat the defect.

Also records the init state transition (D15) in the data model.

Related #971
…wing in Complexity Tracking

R13 replaces the spec's literal "rebuilt atomically, so no window exposes one
without the other" with a proven safety property, because the two publications
straddle mcp-go's registry and cannot be made one transaction without forking it
or taking the direct listing off SetTools. A plan that quietly weakens a spec
MUST is exactly what review should catch, so the narrowing is now an explicit
Complexity Tracking row flagged for the maintainer's assent at the tasks stage,
not a silent reinterpretation buried in research.

Related #971
…-5 finding)

Round 5 showed D13 still had a hole: name-only GetTool membership cannot detect
that the registered entry and the catalog entry describe different ORIGINS. A
flipped collision winner could get scope-checked against server A while its
handler dispatches to server B, and the describe path still took its tier from
the catalog, so a read-scoped token could obtain a destructive tool's definition
during the window.

Two mechanisms close it, both verified against the tree:
- Colliding display names are now WITHHELD outright rather than first-writer-wins.
  The spec sanctions this (resolve deterministically OR report it), and it is
  strictly safer than today, which is undefined rather than merely
  non-deterministic: buildDirectModeTools appends both and SetTools collapses
  them last-writer-wins over a map seeded by a map iteration of m.clients.
- A name can still change origin without ever colliding (A removed and B added in
  one reconcile). Closed by a description discriminator: filters and the describe
  resolver compare the catalog entry's rendered description against the registered
  mcp.Tool.Description and fail closed on a mismatch. One string comparison, no
  wire change, and it subsumes the same-name schema-change case.

The describe path also takes its tier from the GetTool result's annotations now,
not the catalog. Residual narrows from served-stale to briefly-withheld —
availability, not correctness. Interleaving tests go from four cases to five.

Related #971
…or claim (round-6 finding)

The round-5 draft claimed the description discriminator "subsumes" the same-name
schema-change case. It does not: the direct description is
fmt.Sprintf("[%s] %s", ServerName, Description) (mcp_routing.go:95), so a
schema-only edit leaves it byte-identical, and in deferred mode the appended
signature is lossy by design so a nested-property change can render the same ~.
No field on the registered mcp.Tool encodes the schema in BOTH modes either --
deferred advertises the same {"type":"object"} placeholder for every tool -- and
Tool.Meta is wire-visible, so a hash stamp is closed off by FR-015.

The identity and origin hazards stay fully closed. The schema-only case is now
recorded as an explicit residual rather than claimed fixed, with its blast radius
argued rather than asserted: dispatch is never wrong (the handler validates
against the schema it captured), so an agent acting on a stale definition is
rejected by the pre-dispatch validator with the CORRECT schema embedded and
succeeds on one retry -- the US3/SC-003 self-healing path this feature ships. It
gets its own Complexity Tracking row flagged for maintainer assent, and a sixth
interleaving test that asserts the residual behaves as documented.

Also fixes two stale contradictions the same review surfaced: data-model.md still
labelled byDisplayName first-writer-wins, and R12's signature sketch still had
the builder publishing the catalog.

Related #971
…alog entry (round-7 finding)

The R13 rule 5 discriminator compared "the catalog entry's rendered description"
against the registered mcp.Tool.Description, but the catalog stored only the raw
upstream description -- so the comparison implied a re-render. In deferred mode
that suffix comes from toolsig.Cache, which mutates independently of any direct
rebuild: Warm adds entries at index time and RetainHashes evicts them
(internal/toolsig/cache.go:79, :108). A miss at registration followed by a hit at
filter time, or the reverse, would flip the recomputed string with no generation
change at all, manufacturing a false mismatch that leaves a still-registered tool
unlisted and undescribable until the next rebuild. That is a persistent SC-007
violation -- strictly worse than the microsecond window the discriminator exists
to police.

renderDirectTools now records each entry's exact renderedDescription on the
catalog before publication, and the discriminator compares two immutable
snapshots. Two no-rebuild interleaving tests added: cache miss->warm and
hit->eviction must both leave the tool listed and describable.

Related #971
…e count reads straight

Six cases had accumulated as five-plus-two-plus-a-sixth. Regrouped into
closed-by-design, closed-by-withholding, and the documented residual, with the
two no-rebuild cache cases called out separately as what they are: proof the
discriminator reads the stored value.

Related #971
…on API and the annotation defaults (round-8 findings)

Cross-model review round 8 (two scoped opencode passes) returned five MAJORs and
four MINORs; all nine verified genuine against the tree, none rejected.

D13 rule 3 was wrong twice over. It told the filters and the describe resolver to
derive the permission tier from the REGISTERED entry's annotations via
contracts.DeriveCallWith "exactly as dispatch does". DeriveCallWith takes
*config.ToolAnnotations, while the registered entry carries mcp.ToolAnnotation --
and mcp.NewTool seeds destructive=true on every tool, with each
With...HintAnnotation overwriting only its own field. A read-only upstream tool
therefore registers readOnlyHint=true AND the untouched destructiveHint=true, and
DeriveCallWith gives destructive top priority: the rule would have classified
nearly the whole catalog destructive, hiding it from every read- and write-scoped
token while dispatch still admitted the call as read. The tier now comes from the
catalog entry's upstream-derived requiredPermission -- identical to today's
behavior and to dispatch's -- used behind the rule-5 discriminator. That makes an
annotations-only change a third stated residual, bounded exactly like the
schema-only one: the handler re-derives the tier from its own captured
annotations, so no call is ever mis-authorized, only a listing can be stale.

R13 rule 1 forbids the builder from publishing, but R12 still specified
buildDirectModeTools as returning []ServerTool alone -- leaving the catalog it
built unreachable and RefreshDirectModeTools with nothing to publish. It now
returns (tools, catalog), non-nil on every path including the DiscoverTools error
return.

NewToolWithRawSchema leaves every annotation hint nil where NewTool seeds four
defaults, so copying only the upstream hints would have made full and deferred
modes marshal different annotations for the same tool (FR-004/FR-008). The
deferred renderer now seeds mcp-go's defaults first.

D15 publishes an empty catalog at init and FR-014 guards the reload, which
together retire the nil-catalog unconditional rebuild -- an accidental self-heal
for a dropped first servers.changed. SubscribeEvents is hoisted out of the
listener goroutine into the constructor, ahead of StartBackgroundInitialization.

Also: spec.md FR-017 gains a note pointing at the plan's three narrowings so the
two documents stop contradicting each other (requirement text unchanged);
rule-count drift fixed (five, not three/four) and two mislabelled rule-4
references corrected; the skew matrix is seven interleavings plus two no-rebuild
cache cases, with the annotations-only case added; generation is given its two
concrete uses and a data-model row instead of being vestigial; quickstart no
longer promises a signature on every entry, which contradicted FR-005.

Related #971
…iduals field by field (round-9 findings)

Cross-model review round 9 re-reviewed the round-8 fixes and confirmed all nine,
then found five more. All verified genuine, none rejected.

The residual inventory was incomplete and the narrowing was under-declared. R13
now walks directCatalogEntry field by field and proves exactly three fields can
move while leaving the rendered description byte-identical: paramsJSON,
outputSchemaJSON and annotations. outputSchemaJSON was the one nobody had named,
and it does not fit under the existing schema residual: nothing validates against
an output schema, so the invalid_params self-healing retry that bounds the input
case does not apply. Stated separately, with what actually bounds it -- output
schemas are advisory shape metadata MCP permits to be absent entirely, so a stale
one costs an agent one mis-shaped parse and is corrected at the next publish.

The spec.md FR-017 note declared only the FR-017 deviation, while the
annotations-only residual also relaxes FR-011 and SC-005 (a read-scoped token can
briefly list a tool that just became destructive) and the two schema residuals
relax SC-007. The note now names all of them, and repeats that call-time
authorization is never stale -- the handler re-derives the tier from its own
capture, so the call is refused even while the listing is not.

Also: the contract still told implementers to copy annotations onto the
raw-schema tool "explicitly", which is precisely the instruction round 8 found
broken -- it now carries the seed-defaults-then-override rule. The publisher
count was ambiguous (D12 said two builder call sites, D15 implied a second
independent publisher); pinned to ONE publisher, RefreshDirectModeTools, which
the initial rebuild calls rather than duplicating. Quickstart no longer claims
the placeholder schema on every entry -- describe_tool keeps its real one.

Related #971
…nto source and derived fields (round-10 finding)

Round 10 confirmed the four other round-9 fixes and found the fifth over-claimed.
R13's field walk concluded "nothing else on the entry can skew silently" after
enumerating only the source fields; it skipped requiredPermission, which is a real
directCatalogEntry field that moves without moving the rendered description. No
new authorization hole -- requiredPermission is derived from annotations, whose
residual already covers the behavior -- but the proof as written was invalid, and
an invalid proof is worse than none when it is the thing licensing three declared
residuals.

The walk is now a two-part table. Source fields carry the visibility verdict:
identity and description are visible by construction; paramsJSON is not reliably
visible (never rendered in full mode, and a semantically-equal edit renders the
identical Sig in deferred mode); outputSchemaJSON and annotations are never
rendered at all. Derived fields are shown to add no independent axis: hash covers
server, tool, description and both schemas but not annotations
(hash.ToolHashWithOutputSchema), and requiredPermission is a pure function of
annotations. Three independent source classes, which is exactly what the skew
tests drive.

Round 10 also confirmed the single-publisher pin is safe at construction time
(DiscoverTools snapshots the manager under a read lock and skips disconnected
clients) and judged the design converged, with no decision worth cutting.

Related #971
opencode (gpt-5.6-sol) reviewed the Phase 4 commits. Three findings were real
defects, one was a real design smell, and the rest resolve to documented,
already-accepted properties.

Fixed:

- **Two caller ids naming ONE tool.** "gh__read_file" and "gh:read_file" are
  both valid on this surface, so a batch may legitimately contain both. The
  planner gated the second as not_found — answering "gone" for a tool the caller
  had just been listed. They now share one deduped evaluator ref and each gets
  that ref's verdict back under its own id.
- **Display and canonical namespaces can overlap.** One string can be entry A's
  display name and entry B's canonical id: server "x" + tool "y:z" displays as
  "x__y:z", and server "x__y" + tool "z" canonicalizes to the same. Resolution
  prefers the display map, so the canonical id silently answered with the wrong
  tool's definition. The catalog now withdraws the CANONICAL form of such an id
  — not the display form, which is unambiguous within its own key space and
  whose withdrawal would unlist two working tools to settle a question about a
  third id form.
- **A server name containing ':' was mis-evaluated in check mode.** preflight
  ids split on the first colon, so canonicalizing handed the evaluator a
  different (server, tool) pair and it answered confidently about another tool.
  Check mode now refuses such ids as not_found; they stay listed and fully
  describable, since definition mode reads the snapshot and never canonicalizes.
  A wrong verdict is worse than no verdict. Recorded as a known limitation: it
  is a property of the repo-wide canonical id grammar, not of this feature.

Also fixed, found while answering the reviewer's questions:

- ONE catalog snapshot is now pinned per describe/check call, instead of each id
  re-loading the pointer. A rebuild mid-request could otherwise answer two ids
  of one batch from different generations, and the check planner and its own
  suggestion corpus from different ones again.
- The evaluator's index reader holds a resolved slice rather than a catalog plus
  a storage-backed predicate re-run per entry per ToolsByServer call.
- restore() no longer reverse-scans a map to recover each id's canonical form.

Not changed, with reasons:

- output_schema does reach the indexed surface. That is the contract ("all
  surfaces", D2) and gate 4 passes unregenerated, which T048 verified rather
  than assumed.
- did_you_mean formatting differs between a gated and an evaluator-produced
  not_found (both grammars vs canonical only). Not a disclosure: both draw from
  the SAME session-visible snapshot, so neither can name anything the caller
  could not list.
- Listing↔describe parity across the SetTools-then-publish window. That is D13's
  documented one-directional guarantee, accepted at T001, and it fails in the
  safe direction.

Each fix carries a test that fails without it; the namespace-overlap and
alias-batch fixes were mutation-checked by reverting them.
@Dumbris Dumbris changed the title feat(102): schema-deferred direct mode — Phases 1–3 (config axis, catalog, deferred enumeration) feat(102): schema-deferred direct mode — Phases 1–4 (deferred enumeration + describe_tool on the direct surface) Aug 27, 2026
Phase 5 (US3). A deferred listing advertises `{"type":"object"}`, so an agent
guesses its arguments from a compact signature. This bounds the cost of guessing
wrong at exactly one retry instead of an unbounded debugging loop against an
opaque upstream error.

Live, end to end: omitting a required argument now answers with the FULL stored
schema plus a hint naming the caller's own id; the corrected retry succeeds; and
both calls appear in the activity funnel — the rejection as `error` carrying its
reason, the retry as `success`.

- Validation reads `entry.ParamsJSON`, the STORED upstream schema, never the
  placeholder the listing advertised — a permissive placeholder would accept
  everything and validate nothing. It is mode-independent, so a full-mode client
  that guessed wrong gets the same help.
- It validates the CALLER's args, not the enriched ones: injectAuthMetadata adds
  properties the upstream schema never declared, and a schema with
  additionalProperties:false would reject them — turning our own bookkeeping into
  the agent's bug on a call whose arguments were correct.
- Validation and its activity emission land together, as T064 insists. Landing
  the first alone would start rejecting calls that appear nowhere in the funnel,
  which is precisely the blind spot issue #969 established this handler must not
  have. markSessionWorked stays after it: a call rejected for bad arguments did
  no upstream work, so it earns no durable work session, like the policy blocks
  above it.
- Fail-open is inherited from the validator (FR-013b) and counted, so an
  uncompilable schema never blocks a call a schemaless proxy would have allowed.

T019a was ticked in Phase 2 but never done — makeDirectModeHandler still took
(serverName, toolName, annotations) when Phase 5 arrived looking for the
captured entry, which is the exact failure that task was written to prevent. It
is landed here: the handler closes over its own *directCatalogEntry, so a
request in flight keeps the definition it was dispatched under even as the
catalog is swapped underneath it (R9). T020's tick had the same problem; the
four call sites are updated now.

Verification: internal/{server,runtime} green under -race with the CI skip
regex; golangci-lint v2 clean repo-wide; server-edition build OK. Removing the
validation fails five tests. The enriched-args guard was mutation-checked three
times before it was genuinely falsifiable — the first two versions passed for
the wrong reason (no auth context to enrich with, then a token whose empty
allowed_servers blocked the call before it reached the validator).
…modes

opencode (gpt-5.6-sol) reviewed Phase 5. One finding, and it was right: the
"in BOTH serialization modes" subtest set p.config.DirectToolResponseMode and
then built the handler by hand — but the mode is read inside renderDirectTools
and nowhere else, so both subtests ran the same code and T059's cross-mode claim
was vacuous.

Each subtest now dispatches through the handler the REAL render path produced
for that mode, and the test additionally asserts the two modes advertised
DIFFERENT schemas on the wire (`{"type":"object"}` vs the full one). Making the
renderer ignore the mode now fails it.

Its other seven answers found no defect: placement relative to the policy gates
and markSessionWorked is correct with no double-start path; validating `args`
while emitting `enrichedArgs` matches the call_tool_* path; the captured entry
makes a concurrent rebuild unable to substitute another tool's schema; and
fail-open is reachable and correct.
@Dumbris Dumbris changed the title feat(102): schema-deferred direct mode — Phases 1–4 (deferred enumeration + describe_tool on the direct surface) feat(102): schema-deferred direct mode — Phases 1–5 (deferred enumeration, describe_tool, self-healing calls) Aug 27, 2026
Phase 6 (US4). direct_tool_response_mode is now flippable on a running proxy,
and the concurrency story is measured rather than asserted.

Verified live in one process: editing the config file flipped gh__create_issue
from a full schema to `{"type":"object"}` plus its compact signature, catalog
generation 2 -> 3, no restart. An unrelated edit (tool_response_limit) in the
same session rebuilt nothing.

- T065/T066: the detector clause, deliberately SEPARATE from tool_response_mode.
  Folding them would make an operator's edit to retrieve_tools rebuild the
  direct listing and notify every connected client for nothing. Without a clause
  at all the apply computes empty ChangedFields and is swallowed as "no changes
  detected", so the guarded rebuild below is never even reached.
- The catalog now records the mode it was RENDERED with, and renderDirectTools
  reads that stamp instead of re-resolving config. Only the snapshot knows what
  connected clients were actually served; config only says what the operator
  wants. The failure paths stamp it too, so a flip made while discovery is down
  is not lost (T069).
- T068: the reload rebuild is guarded on real drift. SetTools re-registers
  everything and emits notifications/tools/list_changed to every initialized
  session, so an unguarded call would make any config edit look, to a client,
  exactly like the tool set changing. A nil catalog is deliberately NOT drift —
  there is no "what we served" to compare against, and treating it as drift
  would rebuild on every unrelated reload.

T070's measurements corrected a claim in D13. "The filters deny it, which is
safe" holds for SCOPED sessions; filterDirectModeToolsForAuth short-circuits for
a session with no agent token and no active profile, so an unscoped one is
served the raw registry and never consults the catalog. So an added name is
listed-but-undescribable for an unscoped session during the window, and a
removed name stays describable-but-unlisted until the publish. Neither leaks —
the first is a session already entitled to the whole surface, the second is a
definition that session was served a moment earlier — and both close at the
publish. The comment in RefreshDirectModeTools now says what the window does
instead of what it was hoped to do, and tasks.md records it beside the
checkpoint.

The three accepted residuals are reproduced as specified, including the
signature-identical nested-schema edit (a canonicalization-equal edit will not
do: hash re-marshals the parsed schema). The annotations-only case proves its
compensating property — a read-scoped token still sees the stale listing, but
its CALL is refused, because dispatch re-derives the tier from the annotations
its own registration captured and never reads the catalog.

Verification: internal/{server,runtime} green under -race with the CI skip
regex; golangci-lint v2 clean repo-wide; server-edition build OK.
@Dumbris Dumbris changed the title feat(102): schema-deferred direct mode — Phases 1–5 (deferred enumeration, describe_tool, self-healing calls) feat(102): schema-deferred direct mode — Phases 1–6 (79/89 tasks; Polish remains) Aug 28, 2026
The skew suite stages the publication window by hand, because DiscoverTools has
no injection seam (R12). That left the one thing it could not show: whether the
REAL publisher does SetTools before publish. Once a rebuild completes both land
consistently whichever order they went in, so inverting them failed nothing — I
checked, and a postcondition test over the finished state passes either way.

D13 rule 1 calls that order load-bearing, so it should not rest on a comment.
This adds a three-line seam (directRebuildPause, nil in production, written only
by tests) invoked between the two publications, and a test that observes the
real RefreshDirectModeTools mid-flight: the registry is replaced while the
catalog generation is still the previous one. Inverting the order now fails it.

Also adds the postcondition test the seam does not replace: every registered
upstream name is admitted by the catalog published with it, and vice versa.
opencode (gpt-5.6-sol) reviewed Phase 6. Six findings; five were real, and three
of them were tests that proved less than they claimed.

Production fixes:

- **Check-then-lock race in the reload guard.** directSerializationDrifted() was
  read outside directRefreshMu, so two concurrent reloads could both observe
  drift, serialize inside the rebuild, and publish two generations — the second
  justified by no flip at all, pushing a second tools/list_changed to every
  client. That is the churn the guard exists to prevent, reintroduced by the
  guard's own racy read. The check and the rebuild it authorizes are now one
  critical section (refreshDirectModeToolsLocked).
- **"" and "full" reported as a changed field.** They are the same mode, so
  deleting the key — or a PATCH round-trip dropping it — claimed a change that
  did not happen. The rebuild guard suppressed the churn, but the apply RESULT
  still misreported it to the caller. Compared normalized now.

Test fixes, each mutation-checked:

- **The origin-flip test was vacuous.** It flipped alpha__run to beta__run —
  different display names, so not an origin flip at all, and its loop skipped
  every unresolved tool and dispatched nothing. A real flip needs the "__"
  ambiguity: "a__b__c" is (a, b__c) or (a__b, c). Rewritten as that, and it now
  asserts the thing that matters — during the window the stale catalog scope-
  checks against the OLD origin and still lists the name, but the REGISTERED
  handler refuses the call naming the NEW origin, because dispatch re-derives
  authorization from the entry it captured. Disabling that check fails it.
- **The two schema-residual tests ran in full mode**, where a schema change is
  plainly visible in the listing — so "silently stale" was demonstrated by a
  listing that was not silent. They run deferred now and additionally assert the
  wire bytes are IDENTICAL across the change, which is what makes the staleness
  silent. Putting them back in full mode fails them.
- **The annotations residual built its own handler** rather than using the
  registered one, so it proved only that a handler the test constructed refuses.
  It dispatches through p.directServer's actual handler now.

Not changed, documented instead: a flip made while DiscoverTools is failing
publishes an empty catalog stamped with the new mode (T069 requires exactly
that), so the guard afterwards sees no drift and will not retry. Deliberate —
the empty listing is itself the visible problem, and the servers.changed that
fixes discovery rebuilds unconditionally. Making the guard retry would rebuild
on every reload for as long as discovery stayed down.
…73-T081)

Five of Phase 7's tasks, four of them produced in parallel by subagents and
integrated here.

- T073/T074: the standalone direct built-in gate. Deliberately NOT an entry in
  toolsListGoldenSurfaces — that list is driven by _DeltaIsEnumerated, which
  reads a frozen pre099/<surface>.json per surface, and direct mode has no
  pre-feature baseline because nothing was registered on it before this spec.
  With zero upstreams the listing is exactly describe_tool, pinned byte-exact in
  BOTH modes together with the instructions string the surface now carries.
  T074's point is real: directServer is the one routing-mode server carrying
  WithToolFilters, so "registered" and "served" are different questions here —
  the gate captures both and fails on any difference.
- T075: the direct_deferred bench arm. Every existing arm encodes a
  retrieve_tools RESULT SET; this is the first to render a tools/list payload,
  which is why the profiler could not price this feature before.
- T076: measured through BOTH paths, and they agree. The profiler says
  direct_deferred saves 29.5% against baseline_json; the in-process gate, which
  measures the real renderDirectTools output rather than an arm's re-encoding,
  says 29.7%. They differ only by the annotations block the arm does not model,
  a constant in both columns. SC-002 is 86.7% against its 80% floor.

  So SC-001's >=70% is confirmed unreachable INDEPENDENTLY of the
  implementation, by the spec's own sanctioned pipeline. The >=70% assertion the
  task asks for is deliberately not added: a test that cannot pass is not a
  gate. This is the one item left that needs a maintainer decision.
- T079/T080/T081: docs/configuration.md, a new docs/features page, and the two
  CLAUDE.md edits. The feature doc's client-compatibility section is the part
  worth reading — schema-driven form UIs render an EMPTY form under deferral and
  should stay on "full".

Also wired the new feature page into website/sidebars.js. Checked the docs-site
trap on the way past: prepare-docs.sh copies docs/ wholesale (no include
allowlist any more), and the new page links to ../configuration/config-file.md,
the PUBLISHED variant — linking the unpublished root configuration.md would
break the site build.

T077 (E2E) is still outstanding and lands next.
Spec 102 is complete: 89/89 tasks.

T077 landed as three E2E tests driving a REAL client over HTTP through the real
POST /api/v1/config/apply path, which is stronger than the in-process version I
had written alongside it — that one only called the rebuild helper and read
ListTools(), so I dropped it:

- SC-006: a client that is already initialized and has already listed the tool
  set sees the NEXT tools/list reflect the flip. The client is deliberately
  never reconnected — a reconnect would prove only that a fresh session is built
  from current config, which is the easy half.
- SC-003: a wrong guess returns invalid_params with the full schema embedded,
  and the retry corrected from that schema reaches dispatch.
- FR-003: /v1/tool_code and /v1/tool-code are asserted against /mcp/all rather
  than against a hand-written expectation, so the test states the invariant (the
  aliases serve the SAME listing) instead of re-encoding the renderer. They are
  the easiest members of the direct-serving set to forget, being registered ~40
  lines from the /mcp/all block and named after a feature that no longer exists.

T083 verified rather than assumed. All five frozen gates pass, and golden churn
across the ENTIRE branch is exactly the two enumerated regens plus four new
files. code_execution_mode.json and both pre099/ baselines are untouched, which
is what confines the change to the two surfaces that register describe_tool.

T082: go test -race ./internal/... is green except
TestResolveDockerStatusResolvableAndWorking, which passes in isolation and on a
re-run of its own package — it writes a fake docker into a TempDir and
manipulates PATH, so it is PATH-sensitive under a parallel whole-tree race run.
It never touches the direct surface. golangci-lint v2 clean repo-wide;
server-edition build OK.
@Dumbris Dumbris changed the title feat(102): schema-deferred direct mode — Phases 1–6 (79/89 tasks; Polish remains) feat(102): schema-deferred direct mode — complete (89/89 tasks) Aug 28, 2026
…ering

Spot-checking the agent-written feature doc by actually rendering its published
examples through toolsig.Render: three of four matched byte for byte, one did
not. The doc showed

  fetch_page(origin*:str, ttl:int=3600, format:enum[json|text], account~:obj)

which reads as declaration order. The renderer emits

  fetch_page(origin*:str, account~:obj, format:enum[json|text], ttl:int=3600)

— required first in required-array order, then optionals sorted BY NAME. The
doc's own next sentence claimed the rendering is deterministic, which made the
wrong example more misleading rather than less: an agent author would have taken
it as evidence that the signature preserves schema order.

Corrected, and the ordering rule is now stated outright instead of implied,
including the negative half (a signature does not tell you what order a schema
declares its properties in).
I wrote that the bench arm's 29.5% and the in-process gate's 29.7% "differ only
by the annotations block the arm does not model, and it is a constant in both
columns". That is not rigorous and the arithmetic does not support it: adding a
constant to both columns LOWERS a savings percentage rather than cancelling, and
the two paths model different field sets anyway — the arm emits canonical JSON
without annotations or mcp-go's _meta, the gate counts the real marshalled
mcp.Tool including both. Full-mode totals over the same 45 tools are 4386 vs
6116 tokens, which is the size of the modelling gap.

The claim worth making is the weaker, true one: two independently-built
measurement paths land within 0.2 percentage points of each other, and both are
less than half of what SC-001 asks for. Corrected in tasks.md and the PR body.
The Phase 7 cross-model review had in fact completed hours before I noticed —
my own `pgrep -f "opencode run"` wait loops matched their own command lines and
polled forever, so I reported "still running" repeatedly. Loops killed; findings
below.

Documentation (the worst of it — these were agent-drafted and I under-checked
them before committing):

- **The feature doc published the unvalidated #971 estimates as measured fact**:
  "roughly 77% of the payload", "~30K tokens", "takes the listing from ~30K to
  ~3.5-5K". Actual measurements are 29.7% (45-tool corpus) and 34.8% (527-tool
  snapshot), with the input schema at 43% of the payload. Replaced with the real
  numbers and a pointer at `go run ./bench/cmd/bench` for readers who need their
  own fleet's figure.
- **"The cost of a wrong guess is bounded at exactly one retry"** was stated as a
  guarantee in both docs. Returning the schema enables correction; it cannot
  guarantee the agent then builds valid arguments or that the upstream succeeds.
  Reworded to the narrower true claim: a wrong guess never reaches the upstream,
  and the response carries what is needed to fix it.
- The empty-form claim named MCP Inspector specifically; nothing here verifies
  any product's behaviour. Re-grounded in the wire shape instead.

Tests:

- **The built-in gate was not byte-for-byte despite saying so.** It compared
  registered against served with assert.JSONEq, which ignores key ordering and
  formatting — exactly the difference it existed to catch — and generated its
  golden from the registration map rather than the served payload. Now byte-exact
  against the SERVED bytes; mutation-checked by reordering keys, which JSONEq
  passed and Equal fails.
- **The two mode goldens were identical**, so the full/deferred subtests pinned
  the same bytes twice, and the comment above the mode test said the modes "must
  differ somewhere" immediately before asserting they do not. Collapsed to ONE
  golden, with the identity asserted outright as the reason one is enough.
- **The notification assertion could pass on a stale frame** — adding and
  indexing the fixture emits its own tools/list_changed, and nothing drained
  them, so the post-flip wait could consume one of those. Drained before the
  flip.
- A bare `prop.(string)` in the retry loop would panic the test binary instead
  of failing an assertion.

CLAUDE.md: line 27 said Go 1.24 while go.mod and line 161 say 1.25.5; the tool
format line named only `<server>:<tool>` though the direct surface lists as
`<server>__<tool>` and accepts both.

Confirmed and already fixed before the review landed: the bench arm's
annotations omission biases savings UPWARD (a ratio's denominator does not
cancel), which its comment had called "cancels in the comparison".
…red-impl

# Conflicts:
#	CLAUDE.md
#	ROADMAP.md
@Dumbris
Dumbris changed the base branch from 102-schema-deferred-plan to main August 28, 2026 17:38
@Dumbris
Dumbris marked this pull request as ready for review August 28, 2026 17:38
…ec 102

None of these were caused by spec 102. All three were found by auditing the
questions the feature raises — "can we tell if anyone turns it on?" and "how
does a user switch modes?" — and all three were verified against a running
proxy, not just read.

TELEMETRY

1. **describe_tool calls were recorded and silently dropped.** It has called
   recordBuiltinTool since spec 085 but was never added to
   builtinToolAllowList, so RecordBuiltinTool's unknown-name early return
   discarded every call and the counter always read zero. This is the counter
   spec 102 depends on most — describe_tool is how an agent redeems a lossy
   signature, so "is the escape hatch used?" decides whether deferral works, and
   an always-zero counter answers it wrongly in the direction that looks like
   non-adoption. The new test enumerates every name the server records, so the
   next tool to record cannot silently fall off the list.

2. **The heartbeat had no field for either serialization axis.** routing_mode
   says which SURFACE is served; it says nothing about whether the operator ever
   turned on the compact or deferred rendering the whole token-reduction effort
   is about. Added tool_response_mode and direct_tool_response_mode as closed
   low-cardinality enums, normalized so "" reports "full" (one bucket, not two).
   Schema 9 -> 10. The getters read the LIVE snapshot via Config(), not the
   legacy r.cfg the older getters use, because both settings are hot-reloadable.

   Two schema pins that were incidental to their tests (the anonymous-id
   stability test, the v9 TPA pin) became floors rather than equalities: pinning
   an exact version forces whoever bumps it next to edit assertions about
   something else, and an edit made to get a green run is how a guarantee
   quietly stops checking anything.

3. **Telemetry's own config pointer went stale on every server add/remove.**
   server_protocol_counts, trust_mode_distribution and the whole feature_flags
   block are built from Service.config, which only moves on
   NotifyConfigChanged — called from ApplyConfig and ReloadConfiguration but NOT
   from SaveConfiguration, the path every server mutation takes. Observed live:
   after adding one stdio server via REST, server_count read 1 while
   server_protocol_counts read all zeros, and stayed there until an unrelated
   file edit forced a reload. Fleets built through the API systematically
   under-reported their own composition.

UX — how a user switches default routing

4. **Changing routing_mode reported success and did nothing.** /mcp binds ONE
   mcp-go server instance at startup from cfg.RoutingMode and registers it on an
   http.ServeMux, which cannot re-register — so the change cannot take effect on
   a running proxy. But routing_mode was absent from DetectConfigChanges
   entirely, so an API apply answered "No configuration changes detected" while
   writing the new value to disk. /api/v1/status, /api/v1/routing, `mcpproxy
   doctor` and the Web UI header then all reported the configured INTENT while
   /mcp kept serving the old surface — four surfaces agreeing on the same wrong
   answer — and the Web UI showed a green success toast, because its warning
   branch keys on requires_restart.

   Now detected and flagged RequiresRestart with a reason naming the cause, plus
   the restart badge in both the Web UI field catalog and the macOS tray one so
   the operator is warned BEFORE saving rather than misled after. The dedicated
   routes (/mcp/all, /mcp/code, /mcp/call) are unaffected — each is permanently
   bound to its own mode by design (Spec 031).

   The test also pins that a routing_mode change does not drag the serialization
   axes into requiring a restart: those genuinely are hot-reloadable, and spec
   102's flip path is the best-behaved of the three.

Verified: go test -race ./internal/... green (bar the known PATH-sensitive
Docker test); golangci-lint v2 clean repo-wide; server-edition build OK; the
macOS tray target builds (swift build).
…roxy

An agent QA'd the finished feature against a real proxy with hostile fixtures —
empty tool names, 50-member enums, rug-pulls — and found three things no unit
test could have. All three are fixed and re-verified live.

1. **DISCLOSURE: an upstream tool with an EMPTY name bypassed both direct
   filters.** Such a tool renders as "server__", which ParseDirectToolName
   rejects because the tool half is empty — so resolveDirectTool classified it
   as a proxy BUILT-IN, and both filters pass built-ins through unconditionally.
   An agent token scoped to other servers could see the name, description and
   annotations of a tool on a server it has no access to. (Dispatch was still
   blocked, so this was disclosure, not execution.)

   The catalog now gets the FIRST say: a name the published snapshot admits is
   an upstream projection whatever it looks like, and only a name the catalog
   does not admit falls through to the structural built-in test. The old
   ordering encoded an assumption the handoff even states — "every upstream name
   goes through FormatDirectToolName, which always inserts __" — that is simply
   false for an empty tool name. Mutation-checked by restoring the old order.

2. **On a first-ever start the deferred listing shipped with NO signatures at
   all, permanently.** The direct surface is rebuilt when a server connects,
   which is BEFORE indexing warms the signature cache — so every entry rendered
   with a Peek miss, and nothing ever rebuilt again. The agent had its schemas
   taken away and got nothing in exchange: strictly worse than full mode. A
   restart hid it completely, because the cache was warm by then, which is why
   T035b's fix (warm on restart) did not cover this.

3. **After a rug-pull the direct surface kept the OLD schema forever**, so
   pre-dispatch validation rejected correct arguments and describe_tool handed
   back the same stale schema — turning the self-healing path into exactly the
   unbounded loop FR-013 exists to prevent, produced BY FR-013.

   2 and 3 are one root cause: applyDifferentialToolUpdate updated the index but
   emitted nothing the direct listener subscribes to, so a tool DEFINITION
   change was invisible to the surface that renders definitions. It now emits
   servers.changed("tools_changed") when something actually changed, guarded so
   an idle discovery sweep still emits nothing.

Verified live, not just by unit test: cold start now renders
"enum50(mode*~:str)" on first listing; after rewriting the fixture's schema the
listing shows "enum50(newparam*:int)" and describe_tool returns
required:["newparam"].

go test -race ./internal/... green; golangci-lint v2 clean; server-edition build
OK.
@Dumbris
Dumbris merged commit 9aef8d6 into main Aug 28, 2026
49 of 50 checks passed
Dumbris added a commit that referenced this pull request Aug 29, 2026
…pec/docs

A 5-lens audit of the branch produced 14 confirmed findings (9 more were
refuted and dropped). This lands the ones that belong in this PR; the four
live-verification findings that are pre-existing and unrelated are filed as
issues #1083 and #1084 rather than widened into this change.

THE REAL BUG — a resolved default WAS writable, which falsifies the claim
this PR made:

  SettingsSection keeps a component-local `dirty` ref, and dirtyKeys is the
  UNION of that ref and the working/original comparison. Nothing keyed the
  section, so Reload replaced both snapshots while leaving the ref populated:
  pick Compact -> Reload -> Save PATCHed `tool_response_mode: "full"` into a
  config that never had the key. Every section is now keyed on a form epoch
  bumped per hydration, so it remounts with an empty ref. Verified live: after
  Reload the abandoned edit is gone, Save is disabled, and the key is still
  absent from the config file on disk.

  This also fixes the general case — the stale ref could strand ANY field.

TESTS THAT COULD NOT FAIL. Cross-model review caught the vacuous
"both copies normalized" assertion in the tray test; the same audit found I
had left the identical tautology in the Web UI twin, plus a Raw-tab test that
matched source text and so could not see whether `cfg` was still the untouched
response by the time it was serialized. Both are now real assertions against
an extracted `hydrateConfigState` helper (the Vue counterpart of the Swift
`hydrate(from:)` seam). Mutation-checked in both directions: normalizing one
snapshot fails them, and stripping the section keys fails the regression test.

SYNC:
- roadmap.yaml: the schema-deferred epic still said `in_progress` with a note
  ending "plan in progress" — spec 102 shipped in #1063. Now `done` with a
  `pr:` ref, and the note's "~88% smaller listing" projection is corrected with
  the measured 29.7%/34.8% and the 38.9% ceiling. ROADMAP.md is GENERATED, so
  the edit is in the YAML and the file regenerated via scripts/gen-roadmap.py.
- spec.md/plan.md asserted SC-001's >=70% as fact in four more places, and
  plan.md claimed "Backend-only; no frontend, tray, or Swift changes" plus
  "no tray state" — both contradicted by the UI work. SC-001 is now marked
  NOT MET with the measurement, the ceiling, and the two options open to the
  maintainer. tasks.md was already honest; only two drift notes corrected
  (T083 said four new goldens, git shows three; T073 named the wrong file).
- docs: "routing_mode sits directly above it" was wrong — tool_response_mode
  is between them. The settings-page restart list named `data_dir`, which is
  not a field at all, and omitted routing_mode.
Dumbris added a commit that referenced this pull request Aug 29, 2026
#1082)

* feat(settings): expose both serialization modes in the Web UI and tray

`tool_response_mode` (Spec 085) and `direct_tool_response_mode` (Spec 102)
shipped with four ways to set them — the config file, an env var, a `serve`
flag and the REST API — and zero UI surfaces. A tray-only operator could not
reach `direct_tool_response_mode` at all, which is the entire point of the
schema-deferred feature.

Both now sit in Settings → General, directly under `routing_mode`, the knob
they qualify. Neither carries a restart badge: the Go change detector appends
them to ChangedFields without setting RequiresRestart, so they hot-apply —
unlike `routing_mode` right above them, which genuinely needs a restart.

Rendering the default honestly needed one new mechanism. Both keys are
`omitempty` on the Go side, where an absent value means "full", so the config
the API returns simply omits them until someone sets one — verified live: a
fresh instance returns neither key. A native <select> (and a SwiftUI Picker)
whose value matches no option renders an empty box, which would have shown the
default as "unset". `SettingField.defaultValue` declares the resolved default
and a normalize pass fills it at hydration, in both clients.

Normalization is display-only, and deliberately narrow:

- Applied to BOTH the working copy and the last-saved snapshot, so an untouched
  field never counts as an unsaved change.
- Never written back. Saving one mode PATCHes only that key; the other stays
  absent from the config file. Verified end to end.
- The Raw JSON tab keeps showing server truth. The Web UI already serialized
  the untouched response; the tray reused `original` for it, so this adds a
  separate `raw` copy rather than letting invented keys leak into that view.
- Only the two fields that opt in declare a default; a test pins that list so
  the pass cannot quietly start inventing values for other unset keys.

`tool_response_mode` had no published documentation — the schema-deferred page
pointed at Search & Discovery for "its own tool_response_mode axis" and that
page never mentioned it. Documents it there and repairs the cross-reference to
a pinned anchor, which also gives the new field a docs link that resolves.

Verified against a real instance on an isolated port and scratch data-dir:
flipping the mode through the Settings UI hot-switched /mcp/all with no restart
— inputSchema became exactly {"type":"object"}, descriptions gained their
compact signature, tool count unchanged, payload 8630 → 6523 bytes.

* fix(runtime,tray): address cross-model review of the settings-mode work

Two findings from a gpt-5.6-sol review, both verified against the code before
fixing.

1. `tool_response_mode` was compared raw in DetectConfigChanges, so "" -> "full"
   reported a change when nothing moved. The direct axis three lines below
   normalizes and documents exactly why ("the apply RESULT would still misreport
   to the caller") — that normalization landed in Spec 102 review and was never
   mirrored to the retrieve axis. Exposing "full" as an explicit choice in the
   Settings form is what makes the explicit-vs-absent round trip reachable from
   the UI, so it matters now. Regression test mirrors the direct-axis one and
   fails against the raw comparison.

2. The Swift "both copies normalized" assertion was vacuous: it compared two
   direct calls to normalizeDefaults, which passes even when ConfigStore
   normalizes only one snapshot — precisely the bug it was written to catch.
   (The Web UI side had a source-level wiring assertion; the tray had nothing.)

   Fixed by extracting ConfigStore.hydrate(from:) from load() and testing the
   real store through it: after hydrating a config that omits both modes, the
   fields resolve to "full", neither is dirty, the form reports zero unsaved
   changes, and the Raw tab does not contain the invented key. A genuine edit
   is still dirty, so the guards cannot pass by dirty-tracking being broken.

   Mutation-checked: normalizing only `working` fails 4 of the new tests and
   none of the old one.

* fix(settings): stop a resolved default being writable; sync roadmap/spec/docs

A 5-lens audit of the branch produced 14 confirmed findings (9 more were
refuted and dropped). This lands the ones that belong in this PR; the four
live-verification findings that are pre-existing and unrelated are filed as
issues #1083 and #1084 rather than widened into this change.

THE REAL BUG — a resolved default WAS writable, which falsifies the claim
this PR made:

  SettingsSection keeps a component-local `dirty` ref, and dirtyKeys is the
  UNION of that ref and the working/original comparison. Nothing keyed the
  section, so Reload replaced both snapshots while leaving the ref populated:
  pick Compact -> Reload -> Save PATCHed `tool_response_mode: "full"` into a
  config that never had the key. Every section is now keyed on a form epoch
  bumped per hydration, so it remounts with an empty ref. Verified live: after
  Reload the abandoned edit is gone, Save is disabled, and the key is still
  absent from the config file on disk.

  This also fixes the general case — the stale ref could strand ANY field.

TESTS THAT COULD NOT FAIL. Cross-model review caught the vacuous
"both copies normalized" assertion in the tray test; the same audit found I
had left the identical tautology in the Web UI twin, plus a Raw-tab test that
matched source text and so could not see whether `cfg` was still the untouched
response by the time it was serialized. Both are now real assertions against
an extracted `hydrateConfigState` helper (the Vue counterpart of the Swift
`hydrate(from:)` seam). Mutation-checked in both directions: normalizing one
snapshot fails them, and stripping the section keys fails the regression test.

SYNC:
- roadmap.yaml: the schema-deferred epic still said `in_progress` with a note
  ending "plan in progress" — spec 102 shipped in #1063. Now `done` with a
  `pr:` ref, and the note's "~88% smaller listing" projection is corrected with
  the measured 29.7%/34.8% and the 38.9% ceiling. ROADMAP.md is GENERATED, so
  the edit is in the YAML and the file regenerated via scripts/gen-roadmap.py.
- spec.md/plan.md asserted SC-001's >=70% as fact in four more places, and
  plan.md claimed "Backend-only; no frontend, tray, or Swift changes" plus
  "no tray state" — both contradicted by the UI work. SC-001 is now marked
  NOT MET with the measurement, the ceiling, and the two options open to the
  maintainer. tasks.md was already honest; only two drift notes corrected
  (T083 said four new goldens, git shows three; T073 named the wrong file).
- docs: "routing_mode sits directly above it" was wrong — tool_response_mode
  is between them. The settings-page restart list named `data_dir`, which is
  not a field at all, and omitted routing_mode.
Dumbris added a commit that referenced this pull request Aug 29, 2026
…inish the ux-audit sweep (#1085)

* chore(roadmap): truth-sync ux-audit and upgrade-nudge

Two epics claimed a state the repo contradicts.

upgrade-nudge was `done` with all four tasks done, but Spec 079 FR-002 —
the human-readable "N releases / M weeks behind" delta required on status,
doctor, the startup log, the Web UI and both trays — was never built. The
four shipped tasks are US-slices and the FR belonged to none of them; spec
079 has only spec.md, so no checkbox surface could catch the gap. It
survived only as TODO(spec-079/FR-002) comments in
internal/updatecheck/checker.go and cmd/mcpproxy/status_cmd.go, which is
how an outside contributor found it (#1081). Epic back to in_progress with
an explicit upgrade-nudge-delta task.

ux-audit was `todo` with both sweeps todo, but both shipped in August
(#1046 Web UI, 36 findings; #1043 macOS tray, 16 findings) along with ten
PRs closing every finding and the five defects the re-check pass raised
(#1061/#1064/#1065, all closed). Recorded the sweeps and fix waves as done
and added the two items that really are open: folding the audit's F6 modal
Escape/focus and F1 cross-view total assertions into e2e/web-ui-sweep, and
verifying the tray fixes on a running tray (the 2026-08-26 re-walk found
the installed tray still on v0.60.0, so #1055 remains unobserved).

Also cleared the two --check-github errors, neither introduced here:

- tpa-db and mcp-2026-upgrade each linked a docs(specs) PR in `pr:`. That
  field is implementation evidence, so a spec merge read as "epic done".
  Both entries' own notes already said implementation had not started;
  moved the refs into the notes and left the statuses honest at todo.
- analytics-default-landing (#1044) and schema-deferred (#1063, tasks.md
  89/89) were genuinely finished and are now done.

* feat(079): "N releases / M weeks behind" upgrade delta (FR-002)

Closes the last outstanding slice of Spec 079. FR-002 requires the
current version, the latest version AND a human-readable delta on
`mcpproxy status`, `doctor`, the startup log, the Web UI and both trays;
what shipped was latest-version-only, and the deferral survived as
TODO(spec-079/FR-002) comments that an outside contributor found (#1081).

The blocker was data, not rendering: the checker fetched only
/releases/latest, so it had the offered release's publish date but not the
running one's, and no tag sequence to count.

internal/updatecheck/delta.go computes the delta as a pure, network-free,
clock-free function, so every spec rule is a table row. Fetching is a
best-effort enrichment hung off the existing check — never a second check
pipeline (FR-001):

- one page of /releases (ETag-conditional, so a repeat costs no rate-limit
  budget), plus a by-tag lookup only when the running build predates that
  page and its date is therefore unknown;
- both requests share one context deadline. A per-request timeout bounds
  each hop and nothing bounds the total, and CheckNow runs synchronously
  inside GET /api/v1/info?refresh=true, so an unbounded delta would stall
  a request a user is waiting on;
- counting is by semver, never by list position: GitHub orders the list by
  created_at, and the live data has v0.53.0-rc.9 sitting ahead of the
  later-published rc.10.

Three degradation rules matter more than the feature. A delta miss must not
set CheckError (statusVersionSuffix returns "" whenever it is non-empty, so
that would DELETE the nudge instead of enriching it), must not touch the
FR-018 backoff counters, and must leave every surface rendering exactly what
it rendered before. There is a regression test per rule.

The clause is rendered once in Go (FormatBehindSummary) and printed verbatim
by every surface, with the raw numbers alongside for machine consumers. That
is deliberate: GuidanceLine is already hand-ported into UpdateBanner.vue with
a "keep the two in sync" comment, and FR-002 asks for consistent framing
across five surfaces.

Fields are additive per FR-021 and had to be threaded through six
hand-maintained mirrors — VersionInfo, the InfoResponseUpdate DTO and its
copier, internal/contracts (→ make swagger), the contracts.ts generator,
status_cmd's map decoding, and the Swift tray's Codable, which turned out
never to have decoded the US2 fields either.

Two adjacent bugs fixed because the delta sits on the same code:

- GetLatestReleaseIncludingPrereleases returned releases[0] from a
  created_at-ordered list, so an RC user could be offered a release older
  than one already published. Now max-by-semver.
- GetReleaseByTag interpolated an unvalidated tag into the request path and
  deferred validation to callers by comment only. The delta feeds it the
  ldflags-stamped version, so it now validates and escapes.

Both trays render the delta in the update item's tooltip rather than its
title: Spec 092 FR-010 fixes the title's exact shape, and a menu row reading
"Update v0.62.0 — ready to restart? — 8 releases / ~14 weeks behind" stops
being scannable.

Known bound, accepted deliberately: the scan reads one 100-release page, so a
build older than that window reports a lower bound ("59+ releases") and a
semver-in-range release created long ago can be missed without setting that
flag. Spec 079's own assumption (L159) allows an approximation where an exact
count is unavailable, and the figure is motivational, not load-bearing.

Verified: unit + integration tests against an httptest GitHub (delta,
saturation, tag fallback, 304 reuse, budget, malformed tags), `go test -race`
across internal and cmd, golangci-lint v2 clean, 962 frontend unit tests, and
a live version-stamped instance confirming the degradation path renders the
pre-delta output when the check fails.

* test(ux-audit): finish the sweep's regression net, and fix what it caught

The August Web UI audit closes by asking for four assertions to be folded
into the committed sweep. Three were already there (F9 contrast, F30
accessible names + aria-live, F14 responsive); this adds the two that were
not, and then fixes the defects they immediately exposed on main.

F6 — the Add Server modal takes focus, traps Tab, closes on Escape and
restores focus to its trigger. These `<dialog :open>` modals are opened by
the `open` attribute rather than showModal(), so none of that comes from the
browser; it all comes from useModalA11y, whose own comment delegates the
real-browser half of its coverage to "the Playwright sweep" — a promise the
sweep did not keep. Escape is dispatched IN-PAGE, never via
page.keyboard.press: re-checking F6 during the audit produced a false
negative for exactly that reason, because a synthetic key never reaches the
document listener under test.

F1 — Activity and Usage must report the same 24h numbers as the API. The
invariant is not naive equality: Usage folds `blocked` into its error tile
while Activity splits them, and /activity/usage is served from a short read
cache while /activity/summary counts live, so the assertion polls for
convergence and encodes the real relationship.

Running the completed sweep turned up three defects already on main, none of
them regressions of the audit's 36:

1. The LANDING PAGE failed AA in both themes (3.09-4.23:1) and two of its
   controls had no accessible name. Usage.vue uses the raw `opacity-*`
   utility for muted text; #1054 fixed that failure mode for the
   text-base-content ramp and .stat-* but never covered raw opacity, and
   nothing caught it because the audit walked `/` before #1044 made Usage
   the landing page. Moved onto the AA-checked --tone-muted; both selects
   got an aria-label.

2. The Activity status pill clips at 390px for any status outside the known
   four. An unmapped status falls through to a ghost pill carrying the raw
   value, and `tool_auto_approved` measured 145px inside a 93px cell (the
   table is table-fixed below 640px). The pill now truncates against the
   cell and keeps the full value in its title, rather than the column being
   sized to today's longest label — which would hold only until the next
   status is added.

The phone check that finally caught #2 had been passing vacuously: it
measures rows.first(), and a `success` row — ~90% of traffic — renders an
sr-only label of zero width. It only became a real assertion once the new F1
check generated an error row and put one first. That coupling is worth
knowing about: the sweep shares one instance, so one check's traffic is the
next check's data, and a newly-failing neighbour after adding a check is more
likely a real find than a flake. Both were separated from the change under
test by running the same sweep against clean HEAD.

Sweep: 25 passed / 3 failed on clean HEAD, 30 passed / 0 failed now.

Roadmap: ux-audit was still carrying both sweeps as `todo` although they and
every finding shipped in August; recorded here along with what the completed
net found.
Dumbris added a commit that referenced this pull request Aug 29, 2026
The Unreleased section is hand-written, and the flagship feature of the next
release was missing from it entirely: schema-deferred direct mode shipped in
#1063 with no changelog entry, and the settings UI in #1082 likewise.

The feature entry states the two things a reader most needs and would otherwise
get wrong: that it is OPT-IN (default `full`, unchanged behaviour), and that the
measured saving is ~30% — not the ~88% the roadmap projected until today. Any
launch copy drafted from the old number would be wrong.

Also records #1083 and #1084 under Bug Fixes.
Dumbris added a commit that referenced this pull request Aug 29, 2026
…; restate SC-001 (#1086)

* fix: direct-surface describe ids (#1083), config-apply errors (#1084); restate SC-001

Closes #1083. Closes #1084.

#1083 — describe_tool on the DIRECT surface returned identifiers that surface
cannot be asked for: the canonical "<server>:<tool>" (it registers
"<server>__<tool>") and a `call_with` naming call_tool_read/write/destructive
(intent variants the direct surface does not expose). Following either verbatim
gave "tool not found". Deferred mode makes this the live discovery path, since
it tells agents to recover schemas through describe_tool.

  applyDirectSurfaceIdentity sits at the same assembly seam as output_schema —
  putting it in the shared builder would change indexed-surface bytes. It swaps
  the registered display name in and drops `call_with`. Dropped rather than
  retargeted: it answers "which variant do I dispatch through", a question this
  surface does not have. No signal is lost — DeriveCallWith is a pure function
  of `annotations`, which travels on the same definition, so a consumer can
  recompute the tier. The D10 safety test is re-pointed at that source rather
  than deleted. Both id spellings still WORK as input; only the echo changes.
  Verified live: the identifier describe_tool returns now executes.

#1084 — three defects:
  1. A value the validator rejects returned 500. ApplyConfig discarded its
     structured errors, so handlers could not tell "you sent a bad value" from
     "we failed to persist a good one" without string-matching. The errors now
     ride on the result and writeApplyConfigError classifies: 400 with the
     offending field in a structured payload, 500 reserved for real faults.
  2. internal/httpapi/server.go called SugaredLogger.Error(msg, "key", val),
     which CONCATENATES — hence "patcherrorconfiguration...". 97 call sites
     converted to the *w variants, including ones passing zap.Field values to
     the sugared API (broken a second way). Argument-parity checked per call.
  3. /api/v1/status embedded a status snapshot whose upstream_stats and
     tools_indexed nothing refreshed — stuck at "Connecting"/0 for the life of
     a process whose servers were long Ready, disagreeing with the correct
     top-level field.

     Cross-model review caught two defects in my first attempt: refreshing
     inside Server.GetStatus() made every poll walk all servers TWICE, and the
     SSE stream forwards a snapshot that never passes through GetStatus() at
     all, so it stayed stale. Now a withLiveUpstreamStats helper at the API
     layer, applied at all three emission sites, each reusing the traversal it
     already does for its top-level field. It copies rather than mutates — the
     SSE map is owned by the publisher.

SC-001 restated per corpus shape (maintainer decision). The original ≥70%/≥85%
was unreachable: measured 29.7% (45-tool corpus) and 34.8% (527-tool snapshot)
against an arithmetic ceiling of 38.9%, because the target assumed schemas
dominate the payload and names/descriptions/annotations actually do. Now ≥25%
and ≥30% — roughly 15% relative headroom below each measured value, as a
regression floor rather than a second projection. Both bounds are asserted, so
that test file IS the SC-001 gate; a new fleet-scale case independently
reproduces T035a's numbers from its own loader. The original 70% is retained as
an upper tripwire: clearing it means the corpus premise changed.

* docs(changelog): record spec 102, the settings UI, and these fixes

The Unreleased section is hand-written, and the flagship feature of the next
release was missing from it entirely: schema-deferred direct mode shipped in
#1063 with no changelog entry, and the settings UI in #1082 likewise.

The feature entry states the two things a reader most needs and would otherwise
get wrong: that it is OPT-IN (default `full`, unchanged behaviour), and that the
measured saving is ~30% — not the ~88% the roadmap projected until today. Any
launch copy drafted from the old number would be wrong.

Also records #1083 and #1084 under Bug Fixes.
Dumbris added a commit that referenced this pull request Aug 31, 2026
…1137)

* docs(specs): 103-token-bench — measured savings, published results

Spec for the token-bench epic, unblocked when spec 102 shipped in #1063.

The premise: MCPProxy's "massive token savings" claim currently rests on a
static tool-surface measurement. That number is real and CI-gated, but it is
not what a user pays. A mode that halves the menu and doubles failed attempts
costs more, not less, and nothing in the repo measures that.

Two facts make it urgent. Spec 102 projected ~70% and measured 29.7% on the
frozen 45-tool corpus, concluding that names/descriptions/annotations rather
than schemas dominate the payload — a conclusion drawn from two frozen corpora
and never checked against a real fleet. And the session-cost rows in today's
report are computed from literature-derived retry defaults, which the dashboard
honestly labels as estimates; every session-level savings claim inherits that
assumption.

Four user stories, P1 first: measure cost per COMPLETED task and first-call
success on replayed real sessions; let an outsider reproduce the numbers;
confirm or correct the spec-102 payload decomposition; publish.

27 functional requirements, 10 success criteria, no [NEEDS CLARIFICATION]
markers. SC-006 is the one that matters most — a mode that costs fewer surface
tokens but completes fewer tasks must look WORSE in the headline metric, so the
benchmark is required to detect a false saving rather than reward it.

A Scope Boundary table names what the existing harness already provides
(encoding arms, live-builder-derived token counting, response cost, break-even,
retrieval metrics, the provenance-badged report, the frozen corpora) and states
that rebuilding any of it is a defect. The genuinely missing pieces are session
replay, measured success/retry rates, tokens-per-successful-task, crossing the
full three-axis mode matrix, and the payload decomposition.

Three decisions were resolved as documented assumptions rather than
clarification markers: the pinned model is a plan-stage budget decision and the
deterministic half stands alone without it; a task-completion signal is assumed
derivable from recorded sessions, with an explicit fallback, and this is the
assumption most likely to be wrong; and the public suite is specified as a
capability rather than named, because the 2026-08 shortlist needs
re-confirmation against the suites' current code before being committed to.

* docs(specs): 103-token-bench — restructure after cross-model review round 1

Seven findings from the opencode gpt-5.6-sol review, all verified against the
tree and all applied. Two were Critical and changed the spec's premise.

1. Replay cannot answer what the spec asked of it. The activity record carries
   tool calls, arguments, responses, status and work-session grouping — but no
   prompt, no conversation, no model state and no completion oracle. Replaying a
   recording under a different mode therefore cannot reveal how the agent would
   have BEHAVED: which calls it would attempt, whether the first was right,
   whether it finished. The original US1 asked for exactly that.

   Added a binding Replay Boundary section and split US1 into US1 (deterministic
   counterfactual cost recomputation over real workload shape, no model
   involved) and US2 (live agent loop under a pinned model, which owns every
   success and completion figure). FR-004 forbids inferring behaviour from a
   recording; Out of Scope names it as forbidden rather than merely excluded.

2. Reproducibility and privacy were in conflict and the privacy side was absent.
   The body-inclusive export deliberately returns full unmasked values including
   detected secrets — it is the compliance surface, not a browsing one. The spec
   kept session contents out of published artefacts but said nothing about
   protecting them during replay, including transmission to model providers.

   Added FR-006..FR-009 (sensitive by default; never transmitted to third
   parties; flagged records excluded or reduced; inputs live outside the repo
   with a documented deletion step) and SC-009. FR-030 now states that figures
   from private sessions are not independently reproducible and cannot be the
   sole support for a published claim.

Also applied:

3. The headline metric did not penalise worse completion — tokens per completed
   task can improve while completion falls, if tokens fall faster. FR-018
   requires completion rate at equal prominence, FR-019 adds a
   completion-regression threshold that overrides any token saving, and SC-007
   requires demonstrating it with a deliberately degraded mode.
4. SC-001 demanded "every mode combination" while FR-017 requires skipping
   invalid ones; scoped to every valid combination.
5. Core terms were undefined; added a binding Definitions block that also
   separates corrective retries from infrastructure retries.
6. Three FRs restated existing harness capabilities; moved to Inherited
   Constraints (IC-001..IC-004) so they are satisfied rather than rebuilt.
7. The capability axes were not a testable matrix; FR-016 now states them as
   binary conditions over the routing modes where each is available.

32 functional requirements, 12 success criteria, 4 inherited constraints.

The reviewer independently confirmed the spec's current-state claims: nothing in
the harness replays activity exports, retry rates are literature-derived
defaults, first-call success is not measured, the three axes are not crossed,
and the spec-102 figures are accurate.

* docs(specs): 103-token-bench plan — 5-cell matrix, MCPMark, split token accounting

Phase 0 research (5 parallel investigations) and Phase 1 design artifacts.
Research changed three of the spec's assumptions, all toward less work.

THE MATRIX IS 5 CELLS, NOT 12. The three axes are not a free product: each
serialization axis governs exactly one surface. tool_response_mode has a single
consumer inside the retrieve_tools handler, and direct_tool_response_mode is
read only when building the direct listing, so seven combinations are
structurally impossible and become skip-with-reason rows. contracts/mode-matrix.md
enumerates all five valid cells and all seven skips with their causes.

THE MATRIX CROSSES ON ONE PROXY INSTANCE. All three routing-mode servers are
built at startup and all three endpoints stay mounted regardless of config, so a
cell is selected by endpoint URL — not by a config change plus restart. This
turns the matrix from expensive to nearly free.

MCPMARK IS ADOPTED, SHA-PINNED. Both questions the spec left open resolve in its
favour: it emits per-task token usage, pass verdicts and turn counts, and it has
exactly one MCP factory seam, so pointing it at mcpproxy is a single env-gated
branch. The spec's fallback of building an in-repo task set is not needed.
Filesystem (30 tasks) + postgres (21) are credential-free and fully local — the
FR-027-clean core.

Design decisions:
- Extension seam is "new measurement mode" (flag + driver + report block), used
  twice, following the spec-085 flip-gate precedent. No new arm, no new binary.
- New bench/replaycorpus/ sibling of corpusio/ (a session trace is not a corpus),
  plus replay.go, modematrix.go and agentloop.go in package bench.
- Report blocks are additive with no version bump.
- Token accounting stays split: tiktoken for everything deterministic, provider
  usage read off the model response for the live loop, never summed — enforced by
  the harness's existing withhold-rather-than-compute pattern.

One production change is required and it is small: request_bytes/response_bytes
exist on the storage record, documented as measured pre-truncation, but are
absent from the export contract. Adding them turns FR-002 from "exclude every
truncated record" into "annotate it accurately".

Security design (constitution principle IV) rests on three findings: the export
path never masks (masking is wired only into list and detail), has_sensitive_data
is written asynchronously after persistence so it is a best-effort reducer rather
than a guarantee, and bodies are not needed for the headline at all. Hence
bodies-off by default.

Five risks carried forward, three of which are pre-existing defects this feature
would otherwise inherit silently: the report's tokenizer identity is a
document-level singleton that would become false once provider usage lands beside
it; provenance is section-level while FR-013 requires measured and estimated
figures to coexist in one block; the per-arm retry lookup returns 0.0 for unknown
arms, indistinguishable from a measured 0.0; tiktoken fetches its vocabulary over
the network unless a cache env var is set, which defeats SC-004 offline; and
SC-011 has no enforcement beyond gitignore.

Constitution check passes with no violations, so Complexity Tracking is omitted.

* fix(specs): 103 — the 7 extra combinations are redundant, not impossible

Cross-model review round 2 died on tool-permission errors before writing its
findings, but its progress output surfaced two real defects, both verified
independently against the tree and both fixed here.

1. The matrix framing was wrong. The 3-axis product is 12 combinations that
   COLLAPSE ONTO 5 distinct behaviours — the extra 7 are configurable and
   behaviourally redundant, not "structurally impossible". Each is now a
   skip-with-reason row that also names the valid cell it collapses onto, and
   the reason codes distinguish an ignored axis (no consumer on that surface)
   from a genuine override (code_execution forcing full). Verified:
   effectiveToolResponseMode has one consumer at internal/server/mcp.go:1585,
   effectiveDirectToolResponseMode one at internal/server/mcp_routing.go:100.

2. "A cell is selected by endpoint URL, not by a config change" was overstated.
   URL selects the ROUTING-MODE axis only; the two serialization axes still
   require config. Whether those fields hot-reload or need a fresh instance per
   value is now recorded as OPEN and must be settled before /speckit.tasks,
   because it decides whether the matrix crosses on one long-lived proxy or on
   several — a change to the runner's shape.

Note on the review itself: the run auto-rejected its own attempts to read
outside the repo (GOMODCACHE) and hit a ripgrep record-size limit, then exited 0
having printed no findings body. Per standing practice that is a FAILED run, not
a clean review, so it is not counted as a completed round.

* fix(specs): 103 plan — byte counts are not token counts (review round 2)

Nine findings from the re-run cross-model review, all verified against the tree
before applying. One invalidated a load-bearing design claim.

1. HIGH — the privacy design rested on a false premise. It claimed bodies-off
   still yields an accurate cost because pre-truncation byte sizes are recorded.
   But request_bytes/response_bytes are BYTE LENGTHS, and tokenizing requires the
   text, so they can only support an estimate. Worse, internal retrieve_tools
   activity emission carries no byte counts at all, so discovery-response cost is
   unrecoverable from the activity log by any route.

   The bodies-off default survives, for a better reason: TOOL-SURFACE cost — the
   term the modes actually change, and US1's headline — is computed from the
   fleet's tool definitions and the call sequence, not from recorded content, so
   it is fully measured with bodies off. Response cost is the part that needs
   text: estimated with bodies off, measured only with bodies on, and the two are
   never presented interchangeably.

2. HIGH — the proposed SC-011 gate did not work. `git status --porcelain
   bench/results` cannot see ignored files and would stay silent about an
   already-tracked report. Replaced with `test -z "$(git ls-files bench/results)"`.

3. HIGH — the report contract contradicted itself: it required a version bump
   whenever an existing field's meaning changes, then proposed narrowing the
   document-level tokenizer field's meaning without one. Resolved additively —
   the existing field keeps its meaning and each new block carries its own
   accounting_source.

4. The "exactly one consumer" claim for direct_tool_response_mode was false: it
   has three production call sites (listing, drift detection, reload logging).
   All three concern the direct surface, so the 12-to-5 collapse holds, but the
   claim is now stated as one SURFACE, not one call site.

5. Serialization hot-reload was recorded as an open question blocking tasks. It
   is not open: tool_response_mode reads the live snapshot per call and
   direct_tool_response_mode rebuilds on config.reloaded. The matrix crosses on
   one instance with a config apply between serialization cells. Blocker removed.

6. The package placement needed a dependency rule. bench/corpusio imports bench,
   so a bench/replaycorpus mirroring it could not be imported by bench/replay.go
   without a second cycle. Now stated explicitly: either replaycorpus imports
   nothing from bench, or orchestration stays in cmd/bench.

7. Added an explicit compose-don't-reimplement rule for modematrix.go, naming the
   existing pieces it must reach through structural interfaces.

8. Sensitive-data wording overstated: there is no persisted HasSensitiveData
   field; it is derived at export from detection metadata added asynchronously
   after initial persistence.

9. The quickstart contradicted itself on offline operation. Setting the tokenizer
   cache variable names a cache but does not fill one, so a genuinely offline
   reproduction needs it pre-populated.

* fix(specs): 103 plan — scope replay per cell, not per feature (review round 3)

Eight findings, all verified before applying. Two corrected an OVER-correction
made in round 2: fixing the byte-count error, I replaced one absolute claim with
another. Round 3 caught both.

1. "Tool-surface cost carries the headline, so bodies-off works" was too broad.
   It holds for three of five cells, not all five. For direct_full,
   direct_deferred and code_exec, what the mode changes IS the static
   tool-surface payload, computed from the fleet's tool definitions with no
   recorded content — measurable bodies-off. For retrieve_full and
   retrieve_compact, what the mode changes IS the retrieve_tools response body,
   so those two require an explicit bodies-on run and are reported as
   unavailable otherwise, never as zero.

2. "Discovery-response cost is unrecoverable by any route" was false. The
   retrieve_tools handler writes the FULL response to the activity log
   (internal/server/mcp.go:2061-2064), so bodies-on does carry it. It is
   unavailable only in the bodies-off configuration.

3. SC-002 asks for byte-identical replay reports, but the report carries a
   wall-clock generated_at stamp, so the documented diff would always fail. The
   quickstart now excludes that field and the plan records the decision as
   required before the determinism test is written.

4. The chosen suite records input/output/total/reasoning per task and has NO
   cache-read field, so FR-014's cache-read axis cannot come from the suite's
   report. It must be captured in the driver from the provider response or
   declared out of reach for suite-driven runs.

5-7. Round-2 fixes were applied to the contracts but not to their mirrors in
   plan.md and research.md, leaving three contradictions: the tokenizer field
   was still described as needing to be narrowed, the obsolete porcelain SC-011
   gate survived in research.md, and the offline remedy still said "set the env
   var" without the population step. All three now match the contracts.

8. The replay package dependency was left as a choice to settle later while the
   structure already committed to bench/replay.go. Decided:
   bench/replaycorpus imports nothing from bench, defines its own domain types,
   and bench/replay.go imports it. Orchestrating from cmd/bench is rejected —
   it would put measurement logic outside the package holding every other
   measurement.

Also recorded, because the export carries no fleet snapshot: every replay figure
scores a recorded workload against TODAY'S fleet. Internally valid across modes,
but not a historical reconstruction, and reports must say so.

* fix(specs): 103 plan — separate menu cost from complete workload cost (round 4)

Five findings, all verified. The first is the THIRD consecutive round in which a
fix introduced the next round's defect, so the correction this time is structural
rather than another restatement.

1. HIGH — code_exec was wrongly classed as fully measurable bodies-off. Its menu
   is static, but the surface also registers retrieve_tools with a handler
   (internal/server/mcp_routing.go:670-676), and those responses are bodies. The
   artifacts now split TWO different questions that the previous three drafts kept
   conflating:
     - MENU cost (what the agent pays to see the tools): measured bodies-off for
       ALL FIVE cells.
     - COMPLETE WORKLOAD cost (menu plus every response consumed): measured
       bodies-off for the two direct cells ONLY. code_exec is partial;
       retrieve_full and retrieve_compact need bodies-on.
   Stating it as a per-cell table with both columns removes the ambiguity that
   produced the last three corrections.

2. The byte-coverage gap was not scoped precisely. Code-execution sub-calls emit
   both byte counts as zero (internal/server/mcp_code_execution.go:811-818), so a
   truncated sub-call cannot use the byte-length estimate either and must fall to
   exclusion accounting rather than contributing zero.

3. The tokenizer contradiction survived in the implementation tree, which still
   said live_report.go would "scope Tokenizer to deterministic sections" — the
   very narrowing the contract forbids. Replaced with per-block accounting_source.

4. The proposed SC-011 gate was placed in bench.yml, which triggers only on v*
   tags and workflow_dispatch and therefore cannot block a PR. Placement is now
   called out alongside the assertion: a PR-triggered required workflow, or an
   explicit statement that SC-011 is post-merge detection only.

5. "The recording contributes real fleet size" contradicted the adjacent, correct
   statement that no fleet snapshot exists. It contributes recorded CALL SHAPE,
   evaluated against today's fleet.

* fix(specs): 103 plan — replay needs a fleet input and yields deltas, not absolutes (round 5)

Five findings, three High, all verified. Two expose real design holes rather
than wording problems.

1. HIGH — a fleet input is MANDATORY and was missing from the whole design. A
   menu cost is a property of the tool definitions the agent was shown, and the
   activity export carries no fleet snapshot. ProxyToolsForMode returns only the
   built-in proxy tools, not the upstream fleet, so under `direct` the menu is
   unavailable without one. The documented `-replay <jsonl>` invocation could
   therefore compute nothing. Replay now requires pairing with a frozen fleet
   corpus or a live proxy, and a recording-only run is an error rather than a
   degraded run. The quickstart commands are corrected accordingly.

2. HIGH — no cell yields an absolute complete-workload cost bodies-off. The
   previous draft claimed the two direct cells did. Complete workload includes
   every consumed response and that text is absent, so what bodies-off actually
   supports is (a) menu cost per cell and (b) the cross-mode DELTA between the
   two direct cells, whose identical call responses cancel out of the comparison.
   That delta is the honest bodies-off headline; absolutes are unavailable.

3. HIGH — the loader boundary and bodies-on measurement contradicted each other.
   ReplayCall carries no response content and no content may leave replaycorpus,
   yet tokenization was placed downstream in bench/replay.go. Resolved by
   tokenizing INSIDE the loader and emitting counts, so no text ever crosses the
   boundary. The two rules can now both hold.

4. Bodies-on does not recover what a truncated retrieve_tools call actually COST
   the agent: the activity log stores the full pre-truncation response while the
   agent consumed the truncated text. Tokenizing the stored text would OVERSTATE
   the agent's cost, so such calls must be excluded or the truncation re-applied.

5. "Provider usage maps 1:1 to input/output/cache-read" is provider-specific and
   unverified while the provider is unpinned — and is contradicted for the suite's
   own per-task output, which has no cache-read field. Scoped accordingly.

* fix(specs): 103 plan — truncation flag never reaches the activity record (round 6)

One finding, and it is a genuine follow-through on round 5 rather than a new
error: round 5 established that truncated retrieve_tools records must be excluded
or have their truncation re-applied, but nothing in the design could IDENTIFY
one.

wasTruncated is computed in the retrieve_tools handler
(internal/server/mcp.go:2019-2035) and only logged. emitActivityInternalToolCall
(:810) has no truncation parameter, so the flag never reaches the activity
record. A loader would therefore be unable to tell a truncated record from a
complete one, and would silently tokenize the stored full response — overstating
what the agent actually paid, which is the exact failure class FR-002 exists to
prevent.

The plan now names a SECOND backend change (propagate the flag onto internal
tool-call activity) and records the fallback explicitly: exclude every internal
retrieve_tools record from response-cost accounting and report the count. The
fallback must be a deliberate decision, never a silent default.

Findings per round on this artifact set: 7, 9, 8, 5, 5, 1 — converging.

* docs(specs): 103 research — record the truncation-flag gap alongside the byte-fields gap

Completes the round-6 fix: plan.md and contracts/replay-input.md already carry
the second backend change, research.md did not.

* fix(specs): 103 plan — reconcile scope statements with the second backend change (round 7)

One finding. The round-6 fix added a second production change (propagating the
already-computed wasTruncated flag onto internal tool-call activity) but three
earlier statements still said the byte DTO fields were the ONLY production change
and that no request-path code is touched. Both were now false.

Corrected in all three places, including the Constitution Check row for Principle
I: the truncation propagation IS on the retrieve_tools request path, though it
passes a value the handler already holds and adds no computation or I/O. The
blanket-exclusion fallback is recorded as the alternative that removes the
request-path touch entirely, at the cost of coverage — an explicit trade, not a
default.

* fix(specs): 103 plan — post-design re-check contradicted its own scope line (round 8)

One wording finding: the post-design constitution re-check said the design
'otherwise adds no production code' while the paragraph above it documents the
export DTO fields and their projection, which are production code. Restated as
touching no other REQUEST-PATH code, with the off-request-path footprint named
explicitly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants