diff --git a/CLAUDE.md b/CLAUDE.md index a83115580..73cf4d438 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -158,6 +158,6 @@ tail -f ~/Library/Logs/mcpproxy/main.log # main log (macOS; Linux: ~/.mcpproxy/ - **Windows installer**: [docs/github-actions-windows-wix-research.md](docs/github-actions-windows-wix-research.md). **Prerelease** (`next` branch + `v*-rc.*` tags, opt-in, off stable channels): [docs/prerelease-builds.md](docs/prerelease-builds.md). ## Recent Changes +- 102-schema-deferred: Go 1.25.5 module toolchain (`go.mod`), backend-only + existing only — mark3labs/mcp-go v0.57.0 (tool surface), santhosh-tekuri/jsonschema/v6 (Spec-085 validator reuse), zap, stdlib. **No new dependencies.** - 098-tools-preflight: Added Go 1.24 module toolchain (repo builds with local Go 1.25) + existing only — chi (httpapi), bbolt (storage), Bleve (index), zap (logging), Cobra (CLI), swaggo/swag v2 (contract regen). **No new dependencies.** - 097-stored-scripts: Added Go 1.25 (os.Root/Root.ReadFile available — R1) + stdlib only (os.Root). **No new dependencies.** -- 096-batched-call-tools: Added Go 1.24 module toolchain (repo builds with local Go 1.25) + existing only — goja (sandbox), mark3labs/mcp-go (tool surface), zap. **No new dependencies.** diff --git a/ROADMAP.md b/ROADMAP.md index dce01646b..082c2911b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -740,7 +740,7 @@ graph LR | --- | --- | --- | --- | --- | --- | | Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | — | [081-release-qa-gate](./specs/081-release-qa-gate/) | | | Analytics dashboard as default page | In progress | P1 | 25/26 (96%) | [069-observability-usage-graphs](./specs/069-observability-usage-graphs/) | | -| Deferred-schema serialization for the direct tools/list surface (spec 102) | In progress | P1 | — | [102-schema-deferred](./specs/102-schema-deferred/) | | +| Deferred-schema serialization for the direct tools/list surface (spec 102) | In progress | P1 | 3/88 (3%) | [102-schema-deferred](./specs/102-schema-deferred/) | | | Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | | | Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | | | MCP protocol upgrade to 2026-07-28 revision | In progress | P3 | — | [058-mcp-2026-upgrade](./specs/058-mcp-2026-upgrade/) | #1033 | @@ -881,4 +881,4 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—` | [098-tools-preflight](./specs/098-tools-preflight/) | `in-flight` | 26/33 (79%) | | [099-describe-check-mode](./specs/099-describe-check-mode/) | `in-flight` | 9/10 (90%) | | [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — | -| [102-schema-deferred](./specs/102-schema-deferred/) | — | — | +| [102-schema-deferred](./specs/102-schema-deferred/) | `in-flight` | 3/88 (3%) | diff --git a/specs/102-schema-deferred/contracts/direct-deferred-surface.md b/specs/102-schema-deferred/contracts/direct-deferred-surface.md new file mode 100644 index 000000000..edec21f70 --- /dev/null +++ b/specs/102-schema-deferred/contracts/direct-deferred-surface.md @@ -0,0 +1,161 @@ +# Contract: Deferred Direct Surface + +Normative wire-level contract for Spec 102. Where this file and +[spec.md](../spec.md) disagree, the spec wins. Reuses two Spec-085 contracts +verbatim: `specs/085-compact-router/contracts/signature-grammar.md` (signature +rendering) and `.../invalid-params-error.md` (self-healing error body). + +## 1. Deferred `tools/list` entry + +With `direct_tool_response_mode: "deferred"`, each upstream tool entry on any +direct-serving route (`/mcp/all`; `/mcp`, `/v1/tool_code`, `/v1/tool-code` when +`routing_mode:"direct"`): + +```json +{ + "name": "github__create_issue", + "description": "[github] Create a new issue in a repository.\ncreate_issue(owner*:str, repo*:str, title*:str, labels?:[str], milestone~)", + "inputSchema": { "type": "object" }, + "annotations": { "...unchanged from full mode..." } +} +``` + +Rules: +- `name`, annotations, membership, count, and ordering source: identical to full + mode for the same session (FR-008/FR-016). +- `description` = full-mode description (`[server] …`, untruncated) + newline + + the **bare tool name** + the compact signature per the 085 grammar (`*` + required — never elided; `~` lossy collapse). The name prefix is normative and + must be prepended by this renderer: `toolsig.Signature.Sig` is the parenthesised + parameter list only (`"(origin*:str, ttl:int=3600, account~:obj)"`, + `internal/toolsig/signature.go:18`) and carries no name. + `toolsig.Signature.Desc` — the first-sentence prefix Spec 085's compact entries + use — is deliberately NOT used here, because deferred entries keep the full + description. On a signature-cache miss the whole suffix (name + signature) is + absent and the entry is otherwise unchanged (never dropped, never delayed — + FR-005). +- `inputSchema` is exactly `{"type":"object"}` — never literal `{}`, never absent, + never carrying upstream properties/required (strict-client safety, FR-004). + Note the mechanism is normative too: `mcp.NewTool` marshals + `{"properties":{},"required":[],"type":"object"}`, which fails this rule and + re-opens the arg-pruning hazard, so deferred entries use + `mcp.NewToolWithRawSchema`. That constructor takes no `ToolOption`s **and + leaves every annotation hint `nil`**, where `mcp.NewTool` seeds + `readOnlyHint=false, destructiveHint=true, idempotentHint=false, + openWorldHint=true` before any option runs — so the deferred renderer MUST seed + those same defaults and only then apply the upstream overrides. Copying only + the upstream hints marshals a different (or empty) `annotations` object for the + same tool and breaks the rule above (research.md R11). +- `outputSchema` is absent (research.md R2). +- Full mode (`"full"`, the default) is byte-identical to pre-feature output modulo + §3's built-in addition (FR-015). + +## 2. In-band convention (FR-007) + +The direct server's `initialize` result carries `instructions` (today absent on +this server instance), static across both serialization modes, conditionally +phrased. The value is the operator's `instructions` config value when non-empty, +otherwise a **direct-specific default**, followed by a blank line and the +deferral legend. It is deliberately NOT `resolveInstructions`'s built-in +`defaultInstructions`: that text tells agents to "Use 'retrieve_tools'" and to +call `call_tool_read/write/destructive`, none of which exist on this surface +(research.md R4/D11). So an operator's configured instructions are never hidden +here, and the default never advertises a workflow this surface cannot serve. +Reference legend text (final bytes pinned by the direct built-in golden, +captured with an empty `instructions` config): + +> Some tool descriptions end with a compact signature `(param*:type, …)`: +> `*` = required, `~` = collapsed/lossy details. When a signature is present the +> listed inputSchema is a placeholder — flat signatures are directly callable; +> for `~`-marked tools call `describe_tool` with the listed tool name to get the +> full schema. + +## 3. `describe_tool` on the direct surface (FR-009/FR-011) + +- **Definition**: the existing single `buildDescribeToolTool` definition + (surface-neutral prose per research.md R5), listed in BOTH serialization modes, + registered via direct tool-set construction so it survives every refresh + (FR-018). +- **Request**: unchanged shape — `tool_ids` (≤5, or ≤50 with `check:true`), + `check`, `filters`. Accepted id forms on this surface: canonical + `":"` AND direct `"__"`; direct ids resolve through + the registration mapping (first-`__` re-parsing is forbidden). +- **Response**: existing shape (`definitions` + per-id `errors`); definitions + additively carry `"output_schema": {…}` when the tool declares one (all + surfaces). +- **Per-id outcomes** (parity with THIS session's direct `tools/list`): + +| Id state for this session | definition mode | `check: true` | +|---|---|---| +| Listed, ready | full definition | `ready` | +| Listed, but pending / changed / tool-level-disabled (non-agent sessions retain **these** — see the note below) | full snapshot-backed definition — a listed tool is never undescribable | `status:"unavailable"` + the evaluator's real `reason` (see the vocabulary note) | +| On a **server-level** quarantined or disabled server | never listed on this surface (see note) → `not_found` | `status:"unavailable"`, `reason:"not_found"` | +| Omitted from this session's listing — any reason: token server scope, operation-permission tier, profile scope, agent callability, or nonexistence | `not_found` + standard remediation | `status:"unavailable"`, `reason:"not_found"` | +| Removed between list and describe | per-id `not_found`, batch not failed | `status:"unavailable"`, `reason:"not_found"` | +| Malformed id | per-id `not_found` + format remediation | `status:"unavailable"`, `reason:"not_found"` | + +**Check-mode vocabulary note.** Definition mode and check mode do NOT share an +error vocabulary, and this table's check column uses check mode's own. Check mode +answers `describeCheckPayload{verdict, checked_at, request_id, results[]}` with +each result `{id, status, reason?, retryable?, action?, detail?, remediation?, +did_you_mean?}` (`mcp_describe_check.go:69-87`); a non-ready id is +`status:"unavailable"` carrying one `preflight.Reason`, and the reason constants +are `tool_pending_approval`, `tool_changed`, `tool_blocked_by_user`, +`tool_denied_by_config`, `server_quarantined`, `server_disabled`, `not_found`, … +(`internal/preflight/reasons.go:22-38`) — note the `tool_`/`server_` prefixes. +FR-009 requires this shape to survive verbatim onto the direct surface, so the +direct adapter **projects, never translates**: it decides which ids reach the +evaluator (listing parity) and canonicalizes their form (§3 request rules), and +passes the evaluator's `status`/`reason`/`action`/`retryable` through untouched. +The distinctions the existing vocabulary draws — user-blocked +(`tool_blocked_by_user`) versus config-denied (`tool_denied_by_config`), and +server- versus tool-level states — are preserved exactly. A test asserts a direct +check response is byte-compatible with the retrieve-surface response for the same +underlying tool state. + +**Note — server-level vs tool-level states.** The spec's Edge Cases say a +non-agent direct listing "retains tools that are pending, changed, quarantined or +disabled". On this surface that is true only of the **tool-level** states: +`DiscoverTools` drops a server before its tools are ever projected — `if +!snapshot.enabled { continue }` and the quarantined-server skip +(`internal/upstream/manager.go:1128-1138`) — so a tool on a quarantined or +disabled *server* is never in the direct catalog and therefore answers +`not_found` here in both modes, exactly like any other unlisted id. Only +pending / changed / tool-level-disabled entries reach the listing and get the +informative verdict. This narrowing does not weaken FR-011; it is the same +parity rule applied to a state the listing never had. + +No reason code, remediation variant, **suggestion**, or timing may confirm the +existence of an id this session's listing omitted. That explicitly covers both +suggestion channels: `did_you_mean` (check mode) and the definition-mode +case-correction hint. Both must be drawn from this session's direct catalog under +the listing-parity gates, or omitted on this surface — the shared preflight +suggestion corpus is server-level-filtered only and is NOT parity by itself +(research.md R10). Remediation strings on this surface must not instruct the agent +to "re-run retrieve_tools": the shared not-found and malformed-id remediations are +rewritten surface-neutral along with the tool prose (research.md R5). + +Membership on both sides of this table is decided by the same catalog: the direct +listing filters resolve display names through the catalog's registration mapping, +never by re-parsing on the first `__`, so listing and describe cannot disagree for +a server name containing `__`. + +Retrieve_tools-surface semantics are unchanged. + +## 4. Pre-dispatch validation (FR-012/FR-013) + +Direct-mode calls validate against the catalog entry's stored `paramsJSON` (never +the advertised placeholder), in both serialization modes, after the +profile/token/callability gates and before upstream dispatch. Failure → the +Spec-085 `invalid-params-error.md` body (error, `error_type:"invalid_params"`, +tool, one-line hint, full `input_schema`), with the hint's describe reference +using an id valid on this surface. Fail-open for uncompilable schemas (counted, +logged); non-argument failures keep their existing shapes with no schema attached. + +## 5. Rollout (FR-014) + +A `direct_tool_response_mode` change via config hot-reload rebuilds the direct +tool set and (via mcp-go `SetTools`) emits `notifications/tools/list_changed` to +connected direct-surface sessions; the tool set (names, count) is identical across +the flip. A config edit that does not change the effective serialization triggers +no rebuild and no notification. diff --git a/specs/102-schema-deferred/data-model.md b/specs/102-schema-deferred/data-model.md new file mode 100644 index 000000000..66b41327b --- /dev/null +++ b/specs/102-schema-deferred/data-model.md @@ -0,0 +1,161 @@ +# Data Model: Schema-Deferred Direct Mode + +**Spec**: [spec.md](spec.md) · **Plan**: [plan.md](plan.md) + +All entities are in-memory; no BBolt or Bleve schema changes. + +## 1. Direct-surface catalog snapshot (`directCatalog`) — FR-017 + +Immutable per-rebuild record of everything the direct surface exposes. Built inside +the direct rebuild from `upstreamManager.DiscoverTools`, published by atomic +pointer swap; never mutated after publication. + +| Field | Type | Source | Consumers | +|---|---|---|---| +| `entries` | ordered slice of `directCatalogEntry` | sorted DiscoverTools projection | listing rendering (order source) | +| `byDisplayName` | map display name → entry | one pass over the sorted projection; **colliding names are withheld** — neither entry is registered, warning logs both origins (R13 rule 5) | describe_tool direct-id resolution + the listing filters' `(server, tool)` identity; = the registration mapping | +| `byCanonical` | map `server:tool` → entry | same pass | describe_tool canonical-id resolution | +| `serializationMode` | `full` \| `deferred` | live config at build time | FR-014 rebuild guard (compare vs live effective mode) | +| `generation` | uint64 | monotonic counter, incremented once per published catalog | logged on every publish beside entry count + mode; the skew tests assert exactly one increment per paused rebuild and none on a guarded no-op reload, which is what makes "old or new generation?" observable rather than inferred (R13) | + +`directCatalogEntry`: + +| Field | Type | Notes | +|---|---|---| +| `displayName` | string | `serverName__toolName` (unchanged naming) | +| `serverName`, `toolName` | string | original pair the handler closes over — dispatch never parses `displayName` | +| `description` | string | upstream description (pre-`[server]`-prefix) — the source the renderer reads | +| `renderedDescription` | string | the **exact** string handed to `SetTools` for this entry in this generation, filled in by `renderDirectTools`: `[server] …` in full mode, plus the name+signature suffix in deferred mode when `Peek` hit. Immutable once published. This — never a recomputation — is what the R13 rule 5 discriminator compares against `mcp.Tool.Description` | +| `paramsJSON` | string | upstream input schema — validation + describe source | +| `outputSchemaJSON` | string | optional; describe `output_schema` source | +| `hash` | string | Spec-032 SHA-256 — `toolsig.Cache.Peek` key + validator memo key | +| `annotations` | `*config.ToolAnnotations` | listing + `call_with`/permission derivation | +| `requiredPermission` | string | `requiredPermissionForDirectTool(annotations)` over the **upstream** `*config.ToolAnnotations` (`mcp_direct_scope.go:18`) — absorbs today's `directToolPermissions` map and is the tier on **every** path: listing filter, describe resolver, and (re-derived from its own capture) dispatch. It is used only after the rule-5 discriminator confirms the catalog and registry entries are the same generation. It is **never** derived from the registered `mcp.Tool.Annotations`: that is `mcp.ToolAnnotation` (wrong type for `contracts.DeriveCallWith`) and carries mcp-go's `NewTool` defaults — `destructiveHint=true` survives unless the upstream explicitly overrides it, so reading the tier there would classify nearly every tool destructive and disagree with dispatch (R13 rule 3, corrected in round 8) | + +**Invariants** +- Entry and its dispatch handler are built from the same `directCatalogEntry` in + one pass, so a handler always validates against the definition its own + registration advertised. This is handler↔schema atomicity, and it is the only + atomicity the design claims: the `SetTools` call and the catalog publish are + two publications and cannot be one transaction (R13). What holds across them is + a safety property, not atomicity — *no request observes a state less + restrictive than both generations, and no request receives a definition for a + name the registry is not currently serving* — delivered by R13's **five** rules + (`SetTools` first then publish, the builder returning the catalog so the + publisher can swap it; filters deny on catalog miss; the tier taken from the + catalog entry's upstream-derived `requiredPermission` behind the discriminator; + describe requires `directServer.GetTool(name) != nil` as well as catalog + visibility; and no display name ever denotes two origins). +- Membership is decided by this snapshot, not index presence: listed ⇒ describable + (definition mode) and validatable; unlisted ⇒ `not_found`, even if indexed. +- Collisions (`a` + `b__c` vs `a__b` + `c`): **both entries withheld**, warning + logs both origins — an ambiguous display name is never served, so listing, + describe and dispatch agree by construction. Not first-writer-wins: that still + lets one name denote different origins across a rebuild (R13 rule 5). +- A display name resolved from the catalog is used only after the entry's stored + `renderedDescription` matches the registered `mcp.Tool.Description`; a mismatch + means the two publications disagree for that name, and the entry is withheld + (filter denies, describe answers `not_found`) rather than served from either + side. The comparison MUST use the stored value, never a re-render: the deferred + suffix comes from `toolsig.Cache`, which mutates independently of direct + rebuilds (`Warm` adds entries, `RetainHashes` evicts them — + `internal/toolsig/cache.go:79`, `:108`), so a miss at registration followed by a + hit at filter time — or the reverse — would manufacture a false mismatch and + leave a still-registered tool unlisted and undescribable until the next rebuild. + That would break "listed ⇒ describable" persistently, not just within the + publication window. + +## 2. Deferred tool entry — FR-004 + +The `tools/list` rendering of one catalog entry under `deferred`: + +| Field | Value | +|---|---| +| `name` | `displayName` (unchanged) | +| `description` | `[server] ` + `"\n"` + `` + `Signature.Sig` when `Peek(hash)` hits; without that suffix on a cache miss (entry still listed — FR-005). `Sig` is the parameter list only and carries no tool name, so the renderer prepends it; `Signature.Desc` is unused here (deferred keeps the full description) | +| `inputSchema` | exactly `{"type":"object"}` — never `{}`, never absent, no upstream properties/required. **Emitted via `mcp.NewToolWithRawSchema`**, not `mcp.NewTool`: mcp-go's schema marshaller always adds `"properties":{}` and `"required":[]` (research.md R11 / D9) | +| `outputSchema` | absent (stripped — research.md R2); `Tool.MarshalJSON` omits it when unset | +| annotations | byte-identical to full mode. The raw-schema constructor takes no `ToolOption`s **and leaves every hint `nil`**, while `mcp.NewTool` seeds `readOnly=false, destructive=true, idempotent=false, openWorld=true` before options run — so the deferred renderer seeds those same defaults first and then applies the upstream overrides. Copying only the upstream hints would marshal a different (or empty) `annotations` object for the same tool (R11/D9) | + +Full-mode rendering is byte-identical to pre-feature behavior (FR-015) modulo the +appended `describe_tool` built-in. + +## 3. Serialization mode resolution — FR-001 + +- Config: `direct_tool_response_mode` ∈ {`full` (default), `deferred`}; invalid + values fail validation naming the accepted set. +- Precedence: env `MCPPROXY_DIRECT_TOOL_RESPONSE_MODE` > file value; serve flag + applies only when explicitly set (Spec-085 flag pattern). +- Read path: one helper, `effectiveDirectToolResponseMode()`, mirroring + `effectiveToolResponseMode` (`internal/server/profile_resolver.go:57`) — reads + the live snapshot via `p.currentConfig()`, defaults to `full` on empty, and is + the ONLY read site, so the renderer, the catalog's recorded + `serializationMode`, and the FR-014 rebuild guard cannot resolve differently. + Unlike its retrieve_tools sibling it takes no per-call `detail` override: MCP + `tools/list` has no parameters (spec Non-Goals). Resolved at catalog build time + and recorded on the catalog; never affects membership (FR-008). +- `routing_mode` value set unchanged; literal `schema_deferred` → validation error + naming the composition (FR-002). + +## 4. Tool id forms (describe_tool on the direct surface) — FR-011 + +| Form | Example | Resolution | +|---|---|---| +| canonical | `github:create_issue` | `splitServerTool` → `byCanonical` | +| direct | `github__create_issue` | `byDisplayName` (registration mapping; never re-parsed) | + +Both forms resolve to the same entry; per-id error vocabulary on this surface: +`not_found` for any id invisible to this session (no reason codes, no existence +signal); visible ids never error in definition mode (snapshot-backed definition), +and under `check:true` return the informative availability verdict from the shared +preflight evaluator. + +**The catalog is also the resolution source for the LISTING side** (D10): the two +direct tool filters (`filterDirectModeToolsForAuth`, +`filterDirectToolsForAgentCallability`) resolve display names through +`byDisplayName` rather than `ParseDirectToolName`'s first-`__` split — for the +`(server, tool)` identity **and** the permission tier, and only after the +description discriminator confirms the catalog entry and the registered entry are +the same generation. The tier is the entry's own `requiredPermission`, derived +from the upstream annotations exactly as dispatch derives it; it is deliberately +NOT `DeriveCallWith` over the filtered `mcp.Tool.Annotations`, which carry +mcp-go's constructor defaults and would read "destructive" for almost every tool +(R13 rule 3). The residual this accepts — an annotations-only change inside the +window — is recorded in plan §Complexity Tracking and is bounded by dispatch +re-deriving the tier from its own captured annotations. Without the +catalog-backed identity, a server whose name contains +`__` is evaluated as a different (nonexistent) server by the listing and as the +real pair by describe, producing a describable-but-unlisted id — the exact FR-011 +disclosure the catalog exists to prevent. + +**Suggestion surfaces are part of the parity contract** (D10): `did_you_mean` +(check mode) and `suggestCanonicalToolID` (definition mode) must draw only from +ids this session's direct listing would include, or be suppressed on this +surface. The shared preflight corpus (`preflight.visibleCorpus`) filters at the +server level only — no permission tier, no tool-level gate — so it is not +listing-parity by itself. + +## 5. describe_tool definition (additive delta) — research.md R2 + +Existing definition object `{name, description, inputSchema, server, annotations, +call_with}` gains optional `output_schema` (raw upstream output schema object), +present iff the resolved tool declares one, on all surfaces, added at the +definition-assembly seam (not `buildFullToolEntry`). + +## 6. State transitions + +- **Upstream refresh** (`servers.changed`): full catalog rebuild → `SetTools` + (includes `describe_tool` by construction — FR-018) → mcp-go emits + `tools/list_changed`. +- **Init** (`initRoutingModeServers`, D15/R14): one initial rebuild — a call to + the single publisher `RefreshDirectModeTools`, not a second copy of it — before + the listeners are wired — with no upstreams connected this publishes the built-in + set (`describe_tool`) and an empty catalog recording the effective mode, so the + direct surface is never observed empty and the FR-014 guard always has a mode + to compare against. +- **Config hot-reload** (`config.reloaded`): rebuild iff live effective + serialization ≠ `catalog.serializationMode`; otherwise no-op, no notification + (FR-014). +- **Signature cache warm-up lag / tool-level quarantine**: entry listed without + signature until the tool is approved and indexed; schema stays reachable via + describe_tool (snapshot-backed) throughout — listing-only degradation. diff --git a/specs/102-schema-deferred/plan.md b/specs/102-schema-deferred/plan.md new file mode 100644 index 000000000..b4eb0220f --- /dev/null +++ b/specs/102-schema-deferred/plan.md @@ -0,0 +1,572 @@ +# Implementation Plan: Schema-Deferred Direct Mode — Full Enumeration Without Schemas + +**Branch**: `102-schema-deferred` | **Date**: 2026-08-25 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `/specs/102-schema-deferred/spec.md` (issue #971, accepted direction 2026-08-13) + +## Summary + +Add a second serialization mode to the **direct surface's `tools/list`**: with +`direct_tool_response_mode: "deferred"` (new dedicated config key, default `"full"`, +opt-in — research.md R1) every visible upstream tool is still listed with its +unchanged `serverName__toolName` name and annotations, but the entry ships the +existing `[server] …` description with its precompiled Spec-085 compact signature +appended and a minimal permissive `{"type":"object"}` input schema instead of the +upstream `inputSchema`/`outputSchema` (~30K → ~3.5–5K tokens per 100-tool listing). +`describe_tool` joins the direct surface (both modes) as the schema recovery stage, +and direct-mode dispatch gains the Spec-085 pre-dispatch validator so a wrong guess +costs exactly one self-healing retry. No new `routing_mode` value: `schema_deferred` +is rejected with a message naming this composition (FR-002). + +Technical approach, grounded in the code: + +- **Catalog snapshot (FR-017) is the load-bearing structure.** A new immutable + `directCatalog` is built inside the direct rebuild from the same + `upstreamManager.DiscoverTools` projection the listing already renders from + (`mcp_routing.go:76` `buildDirectModeTools`), one entry per tool: display name, + `(server, tool)` pair, description, `ParamsJSON`, `OutputSchemaJSON`, Spec-032 + `Hash`, annotations, required permission. Atomic-pointer swapped on + `MCPProxyServer`; it absorbs today's `directToolPermissions` map and becomes the + single source for (1) listing rendering in both modes, (2) signature lookup, (3) + direct-surface `describe_tool` resolution, (4) pre-dispatch validation — the + handler closure captures its own entry's schema+hash at build time, so a + dispatch can never validate against a definition other than the one its own + registration advertised (research.md R9). That covers handler↔schema; the + catalog swap and `SetTools` are still two publications, so the filter/resolver + skew window is bounded separately by R13's **five** rules (`SetTools` first + then publish, filters deny on catalog miss, the permission tier taken from the + catalog entry's upstream-derived `requiredPermission` behind the discriminator, + describe requiring registry membership as well as catalog visibility, and no + display name ever denoting two origins) — a stated safety property, not a claim + of atomicity. The + catalog build iterates a sorted tool list and **withholds** every `__` + display-name collision outright, logging both origins, so an ambiguous name is + never served and dispatch registration, listing and describe resolution cannot + disagree (D13 rule 5; today's code is undefined here, not merely + non-deterministic). +- **Deferred rendering** reuses the Spec-085 signature machinery read-only: a new + non-compiling `toolsig.Cache.Peek(hash)` (research.md R6) serves index-warmed + signatures with observable misses — a miss lists the entry without a signature, + never drops or delays it (FR-005). The permissive placeholder is **not** what + `mcp.NewTool` produces: mcp-go's schema marshaller always emits + `"properties":{}` and `"required":[]`, so deferred entries are built with + `mcp.NewToolWithRawSchema(…, json.RawMessage(`{"type":"object"}`))` — seeded + with mcp-go's own `NewTool` annotation defaults and *then* given the upstream + overrides, because the raw-schema constructor leaves every hint `nil` and + `NewTool` does not (otherwise the two modes marshal different `annotations` + objects for the same tool — FR-004/FR-008). The full-mode branch keeps the + `NewTool` path untouched (verified empirically — research.md R11; mixing the + two on one tool is a hard marshal error, not a silent override). +- **describe_tool on the direct surface** (FR-009/FR-011): registration is + composed into direct tool-set *construction* (`buildDirectModeTools` appends it) + so every `SetTools` refresh keeps it (FR-018) — including the + `DiscoverTools`-error path, which returns `nil` early today + (`mcp_routing.go:81-85`) and would otherwise drop the built-in on the first + upstream hiccup. The built-in is appended on **every** return path, and the + direct built-in golden's zero-upstream fixture is exactly that shape. One `buildDescribeToolTool` + builder still feeds all surfaces — its **five** retrieve_tools-specific strings + (the tool description `:64`, the `tool_ids` `:71` and `filters` `:82` parameter + descriptions, the `describeNotFoundRemediation` constant `:50`, and the inline + malformed-id remediation `:195`) are rewritten surface-neutral, the one-time enumerated golden regen + of FR-010 plus named `describePlainDelta` substitutions (research.md R5). The + `tool_ids` prose must also name both accepted id forms, since FR-011 requires + `server__tool` here. The handler gains a + per-surface resolver seam: existing surfaces keep the index-backed + `toolVisibleToSession` unchanged; the direct surface resolves both id forms + (canonical `server:tool`, and direct `server__tool` via the catalog's + registration mapping — never by re-parsing) through a catalog-backed resolver + with listing-parity visibility: token server scope + **operation-permission + tier** + profile + agent callability. Invisible ids answer plain `not_found` in + both modes; visible ids always get their snapshot-backed definition in + definition mode, and the informative verdict (`pending_approval`/`changed`/…) + under `check: true` via the shared spec-098/099 preflight evaluator, reached + through an adapter that canonicalizes `server__tool` → `server:tool` first + (the evaluator accepts colon ids only) and restores the caller's id and + ordering on the way back (research.md R10, D14). Definitions additively carry `output_schema` when the tool + declares one — added at the definition-assembly seam, NOT in + `buildFullToolEntry`, so full-mode retrieve_tools bytes are untouched (R2). +- **Pre-dispatch validation** (FR-012/FR-013): `makeDirectModeHandler` calls the + existing `p.inputValidator.validateArgs` (`mcp_input_validation.go`) against the + catalog entry's stored `ParamsJSON` — never the advertised placeholder — after + the callability gate and before `CallTool`, rendering the existing + `invalidParamsErrorResult` (full schema + hint) on failure; fail-open per Spec + 085 FR-013b; non-argument failures keep their current shapes. **Exact + placement**: immediately after `directToolCallabilityBlockWithReason` + (`mcp_routing.go:235`) and before `markSessionWorked` — and, matching the Spec-085 + `call_tool_*` path (`mcp.go:2348-2349`), a validation failure emits the + `emitActivityToolCallStarted` + `emitActivityToolCallCompleted("error", …)` pair + so the availability funnel keeps the blind-spot-free property issue #969 + established for this handler. The unconditional + `emitActivityToolCallStarted` at `:246` stays on the dispatch path only. +- **Rollout (FR-014)**: `SetTools` already emits `notifications/tools/list_changed` + to all initialized sessions (verified in mcp-go v0.57.0 — R8), so the new wiring + is one guarded call in the `config.reloaded` branch of + `listenForRoutingModeRefresh` (`server.go:554`): compare the mode the current + catalog was built with against the live effective mode (read via + `p.currentConfig()`, never construction-time `p.config`) and rebuild only on a + real change — an unrelated config edit rebuilds nothing and emits nothing. +- **Convention channel (FR-007)**: static MCP server instructions on the direct + server instance (today it has none — only the default server carries + instructions, `mcp.go:480`), conditionally phrased so they are true in both + modes; the `initialize`-response delta is enumerated and pinned (R4). The value + comes from one new helper — the operator's `cfg.Instructions` when non-empty, + otherwise a **direct-specific default**, then the deferral legend — so the + operator-configurable `instructions` key (`config.go:489`) is not silently + unreachable here (D11) and the fallback never names a tool this surface does + not expose (D16). It is deliberately NOT `resolveInstructions`, whose + `defaultInstructions` advertises `retrieve_tools` and `call_tool_*`. + +## Technical Context + +**Language/Version**: Go 1.24 module toolchain (repo builds with local Go 1.25). Backend-only; no frontend, tray, or Swift changes. +**Primary Dependencies**: existing only — `github.com/mark3labs/mcp-go` v0.57.0 (tool registration, `SetTools` listChanged notification, `WithInstructions`), `github.com/santhosh-tekuri/jsonschema/v6` (already used by the Spec-085 validator being reused), `go.uber.org/zap`, stdlib. **No new dependencies.** +**Storage**: none changed — BBolt and Bleve schemas untouched; the `directCatalog` is in-memory per-rebuild state; `ToolMetadata.OutputSchemaJSON` is already indexed (`internal/index/bleve.go:39`). +**Testing**: `go test -race ./internal/...` (locally: `internal/server` with the CI skip regex); frozen-golden gates in `internal/server` (see Test strategy); E2E in `internal/server/e2e_test.go`; token measurements against the frozen 45-tool corpus via the spec-083/085 profiler pipeline. +**Target Platform**: Linux/macOS/Windows core binary, personal + server editions — pure `internal/`, edition-agnostic (no `//go:build server` code touched; server-edition build verified). +**Project Type**: Single Go project (core server). +**Performance Goals**: Constitution I unaffected — serialization + registration-time work only; signatures come from the index-warmed cache via a pure read (`Peek`), no per-request compilation (FR-005); validation is memoized per tool hash and cheaper than the upstream round trip it prevents. +**Constraints**: FR-015 byte-stability of full-mode direct listings; FR-010 existing goldens pass unregenerated except the two enumerated FR-009 regens; FR-008 identical tool-set membership across modes; FR-013b fail-open validation; SC-001 ≥70% token reduction on the 45-tool corpus. +**Scale/Scope**: ~6 packages touched: `internal/server` (core of the change), `internal/config`, `internal/runtime` (one clause), `internal/toolsig` (one method), `cmd/mcpproxy` (one flag), `bench/arms` (one measurement arm for the SC-001/SC-002 gates); plus docs and generated OpenAPI artifacts. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* — Constitution v1.1.0. + +| Principle | Assessment | +|-----------|------------| +| **I. Performance at Scale** | PASS. BM25 search and indexing untouched. The direct rebuild already runs on upstream changes; deferred rendering *removes* per-entry schema unmarshal work and reads signatures via a pure cache lookup. Pre-dispatch validation is memoized per tool hash (existing validator). | +| **II. Actor-Based Concurrency** | PASS (with one justified primitive). The catalog is written only on the single routing-refresh listener goroutine and read on request paths — an immutable snapshot behind an `atomic.Pointer` swap, replacing the existing mutex-guarded `directToolPermissions` map with the same read-mostly pattern Spec 085 justified for the signature cache. The catalog swap and mcp-go's `SetTools` remain two publications and cannot be made one transaction (mcp-go owns its registry read), so the resulting skew window is closed by ordering + deny-on-miss + registry-membership rules rather than papered over — D13/R13, with an interleaving test. See Complexity Tracking. | +| **III. Configuration-Driven Architecture** | PASS. One JSON config field with env override, serve flag, validation, and hot-reload (FR-001/FR-014); default preserves today's behavior; no tray state. | +| **IV. Security by Default** | PASS. Serialization never changes membership (FR-008/FR-016): all discovery filters run before rendering. describe_tool on the direct surface applies listing-parity visibility *including the operation-permission tier the existing resolver lacks*, and never emits an existence-confirming reason code **or suggestion** for an id this session's listing omitted (FR-011). Parity is enforced on both sides (D10): the listing filters resolve through the catalog instead of re-parsing `__`, and both suggestion channels (`did_you_mean`, case-correction) are catalog-gated or suppressed — closing four concrete parity defects that exist in this tree today — three disclosure paths plus a wrong-safety-hint path (a listed destructive tool describing as `call_with: "read"`) — rather than opening any. Quarantine/approval gates on dispatch are untouched; validation is fail-open, never blocking a call a schemaless proxy would allow. The one place the design accepts staleness — an annotations-only change inside the two-publication window (Complexity Tracking, third row) — touches only listing/describe; call-time authorization is re-derived by the handler from its own captured annotations and is never one generation behind. | +| **V. Test-Driven Development** | PASS. Every behavior lands test-first; the golden gates make surface drift a failing test by construction (see Test strategy). | +| **VI. Documentation Hygiene** | PASS. `docs/configuration.md`, the feature doc under `docs/features/`, `CLAUDE.md` MCP-protocol line, and `make swagger` regen are in scope (see Documentation & wiring). | + +**Result**: PASS. No new dependency, no new package; one new file-local data structure (`directCatalog`) that *removes* a split-source-of-truth defect (FR-017). Re-checked after Phase 1 design: unchanged. + +## Key Decisions (normative for tasks) + +Resolved in [research.md](research.md); recorded here as the plan of record: + +| # | Decision | Where resolved | +|---|----------|----------------| +| D1 | Config = dedicated `direct_tool_response_mode: "full"\|"deferred"`, default `full`; flag `--direct-tool-response-mode`; env `MCPPROXY_DIRECT_TOOL_RESPONSE_MODE`; `routing_mode: "schema_deferred"` rejected with composition-naming message | R1, R7 | +| D2 | Deferred entries strip `outputSchema` too; `describe_tool` definitions gain additive `output_schema` (all surfaces, definition-assembly seam only) | R2 | +| D3 | describe_tool batch cap stays 5 (50 for `check:true`) on every surface | R3 | +| D4 | FR-007 channel = static instructions on the direct server, both modes, conditionally phrased; `initialize` delta enumerated (composition with the operator's `instructions` key: D11) | R4 | +| D5 | FR-009 prose = surface-neutral rewrite of ALL FIVE retrieve_tools strings (description, both param descriptions, `describeNotFoundRemediation`, the inline malformed-id remediation), single builder kept; one-time regen of the two describe_tool-bearing goldens plus named `describePlainDelta` substitutions | R5 | +| D6 | FR-017 = immutable `directCatalog` snapshot, atomic swap, **colliding display names withheld** (not first-writer-wins — D13 rule 5); handlers capture their own schema | R9 | +| D7 | FR-011 = catalog-backed direct resolver with listing-parity gates incl. permission tier; plain `not_found` for invisible ids in both modes; snapshot-backed definitions for visible ids; check-mode verdicts via shared preflight evaluator | R10 | +| D8 | Rollout default = **opt-in, off**; no default flip in this feature (spec Non-Goals; any future flip is its own evidence-gated decision) | spec | +| D9 | Deferred `inputSchema` is emitted via `mcp.NewToolWithRawSchema`, whose `Annotations` are zero-valued — so the deferred renderer seeds mcp-go's `NewTool` defaults (`readOnly=false, destructive=true, idempotent=false, openWorld=true`) and then applies the upstream overrides, or the two modes marshal different `annotations` for the same tool. `mcp.NewTool` cannot produce the FR-004 wire shape and mixing the two is a marshal error | R11 | +| D10 | Parity is closed on every side: the two direct listing filters resolve `(server, tool)` through the catalog instead of re-parsing `__` (the tier comes from that same catalog entry's `requiredPermission`, behind the D13 rule-5 discriminator — never from the registered `mcp.Tool`'s annotations, which carry mcp-go's constructor defaults; D13 rule 3); the definition assembly takes a catalog-supplied annotations override on this surface (`buildFullToolEntry` otherwise reads annotations from the StateView, not from the entry passed in, so a listed pending destructive tool would describe as `call_with: "read"`); and the two suggestion paths (`did_you_mean`, `suggestCanonicalToolID`) are gated by the direct catalog or suppressed on this surface | R10 | +| D11 | Direct-server instructions = operator's `cfg.Instructions` when non-empty + blank line + the deferral legend, so an operator's configured `instructions` is not lost on the direct surface (fallback when empty: D16, not `resolveInstructions`) | R4 | +| D12 | `buildDirectModeTools` splits into a pure `buildDirectCatalog` + `renderDirectTools` pair so the unit matrix can drive a fixture toolset (`upstream.Manager` is concrete and only lists connected clients), and returns **both** the rendered tool set and the unpublished catalog — D13 rule 1 forbids the builder from publishing, so the publisher needs the handle (two call sites: `RefreshDirectModeTools`, one test) | R12, R13 | +| D13 | The catalog swap and `SetTools` cannot be one transaction (`mcp.Tool`'s only free-form field marshals to `_meta`, and mcp-go snapshots its tool map before our filters run), so the plan promises a safety property, not atomicity. Five rules: **`SetTools` first, catalog published immediately after** (which forces `buildDirectModeTools` to return the catalog alongside the tool set — the builder must not publish, so the publisher needs a handle); filters **deny** on catalog miss (built-ins by explicit name set; a *nil* catalog is not a miss); the **permission tier is the catalog entry's `requiredPermission`** — derived from the UPSTREAM annotations exactly as dispatch does, **never** from the registered `mcp.Tool.Annotations`, which carry mcp-go's `NewTool` defaults (`destructiveHint=true` unless overridden) and would classify almost every tool destructive; describe requires `directServer.GetTool(name) != nil` **and** catalog visibility (`GetTool` for membership only, never for annotations); and **a display name never denotes two origins** — colliding names are withheld outright (spec-sanctioned "report it"), and a mismatch between the entry's **stored** `renderedDescription` (captured at render time, never re-rendered — the signature cache mutates independently of rebuilds) and the registered `mcp.Tool.Description` fails closed. Identity/origin hazards are fully closed; the three independent source fields that can move without moving the rendered description — **input schema**, **output schema**, **annotations** (`hash` and `requiredPermission` are derived and cannot move unless one of those does, so they add no fourth case) — remain stated residuals, the first and third absorbed by the handler re-deriving schema and tier from its own capture (one self-healing retry; never a mis-authorized call), the second bounded by output schemas being advisory. `generation` counter (logged per publish, asserted in the skew tests) + eight interleaving tests + two no-rebuild cache tests | R13 | +| D14 | Direct check-mode adapter canonicalizes `server__tool` → `server:tool` before `preflight.Evaluate` (which accepts colon ids only, so direct ids would otherwise all answer `not_found`), gates invisibility itself without consulting the evaluator, **projects the evaluator's `status`/`reason`/`action`/`retryable` through untouched** (check mode's vocabulary is `status:"unavailable"` + `tool_pending_approval`/`tool_changed`/… — never definition mode's), and restores the caller's original id + ordering in the response and the activity record | R10, contract §3 | +| D15 | `initRoutingModeServers` performs one **initial direct rebuild** — by *calling* `RefreshDirectModeTools()`, which stays the single publisher (D12/R12), not a second copy of the ordering — before the listeners are wired; the direct server registers nothing today (`mcp_routing.go:651-653`), so composing the built-in into rebuild construction alone would still leave FR-009 false until the first `servers.changed`, and would leave the catalog nil so the first unrelated reload churns every client. Because publishing an empty catalog at init retires the nil-catalog unconditional rebuild, `SubscribeEvents()` must also be hoisted into the constructor ahead of `StartBackgroundInitialization` (`server.go:301-302`), or a dropped first `servers.changed` no longer self-heals | R14 | +| D16 | One helper `resolveDirectInstructions(custom string)`: custom when non-empty, else a **direct-specific default**, then the legend. Not `resolveInstructions` — its `defaultInstructions` names `retrieve_tools` and `call_tool_*`. The direct default names ONLY what this surface exposes: `server__tool` calling, `describe_tool`, and the ABOUT links — no `upstream_servers`, which is registered by the code-exec and call-tool builders only (`mcp_routing.go:398`, `:507`), never by `buildDirectModeTools` | R4/D11 | + +### Config interaction matrix (D1 applied) + +| Surface / axis | Governing setting | This feature changes | +|---|---|---| +| `retrieve_tools` serialization (default server, `/mcp/call`, `/mcp/p/`) | `tool_response_mode` (full\|compact) + per-call `detail` | Nothing | +| Direct enumeration (`/mcp/all`; `/mcp` + legacy aliases `/v1/tool_code`, `/v1/tool-code` when `routing_mode:"direct"`) | **`direct_tool_response_mode`** (full\|deferred) — identical across all four routes (FR-003), which share ONE `p.directServer` instance (`GetMCPServerForMode`, `mcp_routing.go:896`), so one `SetTools` covers them all | Serialization only; + describe_tool in both modes | +| `/mcp/p/` | retrieve_tools-mode server — **not** a direct route; no profile-scoped direct surface exists or is created | Nothing | +| Code-execution surface | n/a (always full, no describe_tool — Spec 085 rule) | Nothing | +| Agent tokens / profiles | Discovery filters run before serialization in every mode (FR-016) | describe_tool parity gates added (D7) | +| Both axes combined | Independent: `tool_response_mode:"compact"` + `direct_tool_response_mode:"deferred"` is legal and each governs only its own surface | — | + +## Project Structure + +### Documentation (this feature) + +```text +specs/102-schema-deferred/ +├── spec.md # Feature spec (merged; normative) +├── plan.md # This file +├── research.md # Phase 0: D1–D16 resolutions + mechanical findings R1–R14 +├── data-model.md # Phase 1: directCatalog, deferred entry, mode resolution, id forms +├── quickstart.md # Phase 1: build/enable/verify walkthrough +├── contracts/ +│ └── direct-deferred-surface.md # Deferred entry shape, instructions text, describe_tool-on-direct deltas & error parity +└── tasks.md # Phase 2 (/speckit.tasks) +``` + +### Source Code (repository root) — REAL paths + +```text +internal/ +├── config/ +│ ├── config.go # DirectToolResponseMode field + ToolResponseModeDeferred const beside +│ │ # ToolResponseMode (:485); validation beside the tool_response_mode block +│ │ # (:2229); routing_mode "schema_deferred" rejection message (FR-002) +│ └── loader.go # MCPPROXY_DIRECT_TOOL_RESPONSE_MODE env alias (beside :717) +├── runtime/ +│ └── config_hotreload.go # DetectConfigChanges: direct_tool_response_mode clause (beside :157) +├── toolsig/ +│ └── cache.go # NEW method Peek(hash) (Signature, bool) — non-compiling, miss-reporting (FR-005) +├── server/ +│ ├── mcp_direct_catalog.go # NEW: directCatalog type + build (sorted; colliding display names WITHHELD, +│ │ # both origins logged — D13 rule 5) + atomic store on MCPProxyServer, +│ │ # PUBLISHED BY THE CALLER AFTER SetTools, never mid-build; absorbs +│ │ # directToolPermissions +│ ├── mcp_routing.go # buildDirectModeTools splits into buildDirectCatalog + renderDirectTools (D12) +│ │ # and RETURNS ([]ServerTool, *directCatalog) so the publisher can swap the +│ │ # catalog after SetTools (D13 rule 1) — non-nil catalog on EVERY path incl. +│ │ # the DiscoverTools error return (:81-85); +│ │ # render full|deferred via live config (deferred uses NewToolWithRawSchema +│ │ # seeded with mcp-go's NewTool annotation defaults, D9) +│ │ # → append describe_tool registration (FR-018); makeDirectModeHandler: +│ │ # pre-dispatch validation from the captured catalog entry (FR-012/013); +│ │ # initRoutingModeServers: WithInstructions on directServer (FR-007/D11/D16, +│ │ # direct-specific default, NOT resolveInstructions' retrieve_tools text) +│ │ # + the INITIAL direct rebuild the surface has never had (D15, :651-653), +│ │ # done by CALLING RefreshDirectModeTools — one publisher, not two; +│ │ # refresh guard: rebuild-if-serialization-changed (FR-014) +│ ├── mcp_direct_scope.go # filterDirectModeToolsForAuth: resolve (server,tool) through the catalog instead +│ │ # of ParseDirectToolName; tier = that entry's requiredPermission (upstream- +│ │ # derived, same as dispatch) behind the rule-5 discriminator — NEVER +│ │ # DeriveCallWith over the registered mcp.Tool.Annotations, which carry +│ │ # mcp-go's destructive=true default (D10/D13.3); +│ │ # retires lookupDirectToolPermission's separate map, keeps +│ │ # requiredPermissionForDirectTool as the catalog's derivation +│ ├── mcp_direct_callability.go # filterDirectToolsForAgentCallability: same catalog resolution (D10) +│ ├── mcp_describe_tool.go # Surface-neutral prose — all FIVE retrieve_tools strings (D5/R5); resolver seam +│ │ # parameter; additive output_schema at definition assembly (D2) +│ ├── mcp_entry_builder.go # definition-assembly seam: optional annotations override so the direct surface +│ │ # sources them from the catalog, not the StateView (D10); full-mode retrieve +│ │ # path byte-identical (buildFullToolEntry logic otherwise untouched) +│ ├── mcp_describe_direct.go # NEW: catalog-backed direct resolver (both id forms, listing-parity gates incl. +│ │ # permission tier, not_found discipline) + check-mode delegation (D7) +│ ├── mcp_describe_check.go # id-gate seam for the direct surface (D10); verdict logic unchanged +│ ├── preflight_glue.go # NEW directCatalogIndexReader passed as preflight.EvalContext.Index on the +│ │ # direct surface (:103 today builds preflightIndexReader), so id resolution +│ │ # AND the did_you_mean corpus share the catalog authority (D10/FR-017) +│ ├── mcp_visibility.go # serverInScope reused (evaluateToolGate lives in tool_gate.go:80, isToolCallable +│ │ # in mcp.go:6068); suggestCanonicalToolID gains the resolver seam (D10); +│ │ # existing retrieve-surface resolvers byte-identical +│ ├── mcp.go # MCPProxyServer field for the catalog pointer; sigCache handle reuse +│ ├── server.go # listenForRoutingModeRefresh config.reloaded branch (:554): guarded direct rebuild; +│ │ # AND hoist SubscribeEvents() out of the listener goroutine into the +│ │ # constructor, before StartBackgroundInitialization (:301-302), so the first +│ │ # servers.changed cannot be dropped now that D15 removes the nil-catalog +│ │ # accidental self-heal (R14) +│ ├── toolslist_snapshot_test.go # NEW standalone direct built-in gate (NOT added to toolsListGoldenSurfaces) +│ ├── describe_plain_corpus_test.go# describePlainDelta: enumerate the remediation-prose substitutions (gate 4) +│ ├── mcp_describe_tool_test.go # invert + rename the "direct routing mode" subtest of +│ │ # TestDescribeTool_RegisteredInRetrieveToolsModeOnly (:367) +│ ├── mcp_routing_test.go # makeDirectModeHandler signature change (it now captures its catalog entry) +│ ├── toon_surface_isolation_test.go # same signature change (:187) +│ └── e2e_test.go # flip-notification, self-healing-retry, describe-on-direct E2E +cmd/mcpproxy/ +│ └── main.go # --direct-tool-response-mode serve flag (pattern of :141/:769) +bench/arms/ # NEW deferred-direct arm + testdata golden for the SC-001/SC-002 gates +oas/ # regenerated via `make swagger` (config struct change) +docs/ +├── configuration.md # new field + env/flag +└── features/ (schema-deferred doc) # deferral convention, describe_tool on direct, rollout notes +``` + +**Structure Decision**: Single Go project; no new packages. The two new files live in +`internal/server` beside their Spec-085 siblings (`mcp_describe_tool.go`, +`mcp_visibility.go`) because the catalog and the direct resolver are server-surface +concerns with no external consumer — a leaf package (the 085 `toolsig` argument) +does not apply here. `internal/toolsig` gains one method, no structural change. + +## Test Strategy (incl. frozen-golden handling — FR-010/SC-004) + +**The five frozen gates, by name, and how each is handled** (the fourth pins +describe_tool *response* bytes, not tools/list bytes, and was missed by the first +draft of research.md R2; the fifth is a token budget rather than a golden, and is +the one with the least headroom): + +1. `TestMenuSurface_ExactDeltaFromPreFeature` (`mcp_menu_surface_test.go`, + baseline `testdata/tools_list_prefeature.golden.json`, pre-085 capture): + **passes untouched.** describe_tool post-dates its baseline and is already in + its allowed-additions set; this feature adds no tool to the three surfaces it + snapshots and edits no pre-085 tool bytes. Any failure here is a regression. +2. `TestToolsListSnapshot_MatchesMergeBaseGoldens` + `_DeltaIsEnumerated` + (`toolslist_snapshot_test.go`, `testdata/toolslist_goldens/`): the FR-009 + prose rewrite moves describe_tool's pinned bytes on `default_server.json` and + `retrieve_tools_mode.json` → **regenerate exactly those two goldens once**, + deliberately, via the documented `MCPPROXY_WRITE_TOOLSLIST_GOLDENS` flow. + `toolsListAllowedDelta` needs **no edit**: it already enumerates + `describe_tool` on both surfaces (`toolslist_snapshot_test.go:188-192`) from + spec 099. `code_execution_mode.json` and the frozen `pre099/` baseline are + **never regenerated** — the enumerated-delta comparison against `pre099/` is + what proves the change reached exactly as far as claimed. +3. `TestRetrieveToolsFullMode_GoldenByteIdentity` (`mcp_entry_builder_test.go:116`, + byte-exact): it pins **two** goldens, not one — `retrieve_full_default.golden.json` + (subtest `default`) and `retrieve_full_stats.golden.json` (subtest + `include_stats`). **Both pass unregenerated by construction**: + `buildFullToolEntry`'s logic is not modified (`output_schema` lands at the + describe_tool definition-assembly seam only, D2; the D10 annotations override + is an unused optional argument on this path), and describe_tool passes + `toolEntryOpts{}`, so the stats branch is never reached from the new code. +4. **`TestDescribeToolPlainCorpus_ByteIdenticalWithOneEnumeratedDelta`** (`describe_plain_corpus_test.go`, + `testdata/describe_plain_corpus/pre099.json`): replays 18 plain-mode + `describe_tool` calls and pins each **response body** byte-for-byte, allowing + only the substitutions enumerated in `describePlainDelta`. The FR-009 prose + work touches two *response* strings (`describeNotFoundRemediation` and the + malformed-id format remediation — research.md R5), and its doc comment states + that "a reworded remediation" fails. Handling: **extend `describePlainDelta`** + with the named substitutions; the golden itself stays frozen. `output_schema` + is `omitempty` and the current fixture tools declare none, so D2 does not move + these bytes today — a task MUST re-run this gate after the D2 change rather + than assume it. + +5. `TestDescribeTool_DefinitionTokenBudget` (`mcp_describe_tool_test.go:382`): + asserts describe_tool's own definition fits a **250-token budget**, measured + with the pinned encoder the spec-083 profiler uses (spec 099 FR-015). The + budget rose 150 → 250 when check mode added two parameters and the definition + currently measures ~243, so the FR-009 prose rewrite has only single-digit + headroom — and FR-009's whole point is to name BOTH id forms in the + description and the `tool_ids` parameter, which is exactly the kind of edit + that spends it. Handling: measure after the rewrite rather than assume; + if it overruns, shorten the prose first and only raise the budget with the + same deliberate, documented justification the 150 → 250 bump carried. This + gate is not in research.md R2's list either. + +**Existing assertion that must be inverted (not a golden — a hard-coded test):** +`TestDescribeTool_RegisteredInRetrieveToolsModeOnly` +(`mcp_describe_tool_test.go:342`) has a `"direct routing mode"` subtest +(`:367-372`) asserting `describe_tool must NOT be exposed in direct mode (v1)`. +FR-009/FR-018 deliberately reverse that v1 decision, so this subtest is updated +to assert presence (and the test is renamed accordingly). It is green on +`origin/main` today, so its failure during implementation is expected and +enumerated here rather than mistaken for a regression. `TestMenuSurface_*` is +unaffected: it snapshots three surfaces, none of them direct, and compares bytes +only for tools that existed pre-085 (describe_tool is in its `wantAdded` set). + +**New gate (FR-010, mandatory)**: the direct surface's built-ins are currently +unpinnable (its listing is a live projection) — add +`testdata/toolslist_goldens/direct_mode_builtins.json` + a snapshot test that +asserts over **`p.directServer.ListTools()`** — what the server actually serves +after `initRoutingModeServers` — with zero upstream tools (listing = built-ins +only: `describe_tool`) in **both** serialization modes: membership + serialized +bytes byte-exact, plus the direct server's instructions string (D4/D11/D16, +captured with an empty `instructions` config so the bytes are deterministic). It +must read the live server, NOT `buildDirectModeTools()`'s return value: the direct +server registers nothing at init today (`mcp_routing.go:651-653`), so a gate over +the builder would pass while the served surface was empty (D15/R14). A later edit +to either becomes a reviewable golden diff. + +⚠️ **`ListTools()` is the registration map, not the served listing — the gate must +not conflate them.** `MCPServer.ListTools()` returns `map[string]*ServerTool` +straight out of the registry (mcp-go `server/server.go:1032`), while +`handleListTools` serves `filteredTools(ctx)` (`:1767`), which applies every +`WithToolFilter`. `p.directServer` is the ONE routing-mode server that carries +filters — `filterDirectModeToolsForAuth` and +`filterDirectToolsForAgentCallability`, added to `directOpts` and to no other +server (`mcp_routing.go:615-618`) — so on exactly the server this gate targets, +"registered" and "served" can diverge. Left as written, the gate reproduces the +D15/R14 mistake one level up: it would stay green while an over-broad filter +emptied the real listing. Therefore: the built-ins golden asserts over the +registry via `ListTools()` **and** a second assertion drives a real `tools/list` +through the direct server on a session with no agent token, so the filtered +result is pinned too; any difference between the two sets is itself a failure. + +⚠️ **`ListTools()` returns a map, so "byte-exact" needs an explicit order.** Go +map iteration is randomized per run; a golden serialized straight from the map +would be flaky. Sort by tool name before serializing, and say so in the test — +this is also what makes a later diff reviewable. + +⚠️ It MUST be a **standalone** test, not a new entry in `toolsListGoldenSurfaces`: +`TestToolsListSnapshot_DeltaIsEnumerated` reads a frozen `pre099/.json` +for every listed surface (`toolslist_snapshot_test.go:200`) and asserts every +listed surface appears in `toolsListAllowedDelta` (`:228`). Direct mode has no +pre-feature baseline and cannot have one — its built-in did not exist — so adding +it to that slice fails both assertions. + +**Byte-stability (FR-015/SC-004)**: the naive form of this — "copy a test file +into a throwaway worktree of `origin/main` and drive `buildDirectModeTools` with a +fixture toolset" — is **not runnable at the merge-base**, because the D12 pure +seam does not exist there and `DiscoverTools` only returns tools from connected +clients (research.md R12). Two workable forms, in preference order: + +1. **Live stdio fixture** (portable across commits): the capture test connects a + real fixture upstream over stdio, reusing the harness already in this package + (`preflight_e2e_test.go` + `testdata/preflight_fixture_server.js`), then + serializes the direct `tools/list`. That runs unchanged at the merge-base and + on this branch, so the write-env-then-compare procedure + `toolslist_snapshot_test.go` documents applies verbatim. Heavier (needs Node), + so it carries the same build-tag/skip treatment as the other fixture E2Es. +2. **Same-tree differential** (fallback, if 1 is judged too heavy): over one + fixture `[]*config.ToolMetadata`, render through the untouched `NewTool` full + path and through the new `renderDirectTools` in `full` mode, and assert the + marshalled bytes are equal entry-for-entry. This proves the FR-015 property + (full mode is unchanged) without a cross-commit capture, but it cannot catch a + regression that lives in the shared code both sides call — so it is the + fallback, not the default. + +Either way the assertion is the same: deferral-off rendering is byte-identical to +pre-feature output modulo the appended `describe_tool` entry. + +**Unit (test-first, per constitution V)** — the core matrix. All of it drives the +pure `buildDirectCatalog`/`renderDirectTools` pair (D12) with a fixture +`[]*config.ToolMetadata`, because `upstream.Manager` is concrete and +`DiscoverTools` only returns tools from connected clients (research.md R12): +- Deferred entry rendering: signature appended per 085 rules; the marshalled + `inputSchema` is byte-exactly `{"type":"object"}` — asserted on the JSON, not on + the Go struct, since `mcp.NewTool` would marshal + `{"properties":{},"required":[],"type":"object"}` (R11/D9) — no upstream + properties/required; **the marshalled `annotations` object is byte-identical to + full mode's for three fixtures — nil upstream annotations, one hint, all five — + proving the raw-schema constructor was seeded with mcp-go's `NewTool` defaults + (D9)**; cache-miss entry listed signature-less (FR-004/005); a deferred entry + marshals without `errToolSchemaConflict`. +- Set identity full↔deferred: same names, count, annotations, ordering source + (FR-008), incl. under agent-token and profile filters (SC-005, FR-016). +- Catalog: colliding display names withheld in both flattening directions (neither + listed, neither describable, warning names both origins), atomic swap, entry↔ + handler schema capture, `describe_tool` survives `SetTools` refresh (FR-018). +- Direct describe: both id forms resolve identically through the registration + mapping (incl. a server name containing `__`); permission-tier gate (read-scoped + token → `not_found` for a destructive tool); listed-pending tool → definition in + definition mode and `pending_approval` under `check:true` (SC-007); removed + server → per-id not_found without failing the batch; `output_schema` present iff + declared; **a listed pending/changed destructive tool describes with its real + annotations and `call_with: "destructive"`** — the case where the StateView + annotation lookup would otherwise return nil and silently downgrade the safety + hint to `read` (D10); **a `server__tool` id under `check:true` returns its + verdict under the id the caller sent** — the canonicalization + restore path of + D14, which without it answers `not_found` for every direct id. +- **Listing↔describe parity, both directions** (D10): with a server whose name + contains `__`, the listing filters and the describe resolver agree on the same + entry — asserted by comparing the session's rendered `tools/list` name set + against the set of ids that resolve in definition mode, for an admin session, a + read-scoped token, a write-scoped token, and a profile-pinned session. No id may + be describable-but-unlisted (disclosure) or listed-but-undescribable (SC-007). +- **Suggestion discipline** (D10): a read-scoped token's `check:true` miss returns + no `did_you_mean` entry naming a destructive tool absent from its own listing, + and the definition-mode case-correction suggestion is gated by the same + catalog-backed resolver. +- **Instructions** (D11): a custom `instructions` config value still appears on the + direct server's `initialize`, with the deferral legend appended, in both modes. +- **Publication skew** (D13) — a rebuild paused between `SetTools` and the catalog + publish, with a concurrent scoped `tools/list` and `describe_tool`. Eight + interleavings, in three groups: + 1. *Closed by the design* — an added name, a removed name, a same-name + **description-visible** change, and a same-name **origin flip** (server A + removed and server B added in one reconcile, B's tool flattening to A's old + display name). Each must be withheld or served whole, never mixed. + 2. *Closed by withholding* — a within-generation display-name collision: + neither entry listed, neither describable, in both generations, warning + naming both origins. + 3. *The three documented residuals* — an **input-schema-only** change, an + **output-schema-only** change, and an **annotations-only** change (read → + destructive), each with the description and rendered signature held + byte-identical. These are the only three **independent source** fields on + `directCatalogEntry` that can move without moving the rendered description + (R13's field table; `hash` and `requiredPermission` are derived and cannot + move unless one of these does, so they add no fourth case, and the + catalog-level fields are membership or generation state covered by the other + groups). The test set is the proof of that partition, not just of the + behaviors. Constructing the input-schema case takes care, and the recipe is + normative: the edit must be **semantically different but signature-identical** + — a change confined to nested properties, which the 085 grammar collapses to + a `~` marker, so `Sig` renders byte-for-byte the same — **and** the new + hash must be warmed into the signature cache before the rebuild renders, or + `Peek` misses, the suffix vanishes, and the description changes visibly + instead. A canonicalization-equal edit will NOT do: `hash` re-marshals the + parsed schema (`internal/hash/hash.go:95-104`), so such an edit is the same + schema and the validator cannot behave differently, which would make the + self-healing assertion vacuous. The annotations case uses `read → + destructive`, the sub-change the tier is sensitive to. All three assert + accepted + behavior, not a fix: the stale definition/tier may be served for the width of + the window, while **dispatch is never wrong** — the handler validates against + the input schema it captured, so the `invalid_params` error carries the NEW + schema and one retry succeeds (US3/SC-003), and it re-derives the tier from + the annotations it captured, so a read-scoped token's call to the + newly-destructive tool is still refused at the handler. The output-schema + case asserts the one residual with no self-healing path and documents that + its only consequence is a stale advisory `output_schema` in a describe + response. + + Plus two **no-rebuild** cases the discriminator must not mistake for a + generation change: a signature-cache **miss→warm** and a **hit→eviction** + between registration and a later filter/describe call (`toolsig.Cache.Warm` / + `RetainHashes`). Both must leave the tool listed and describable — the proof + that the comparison reads the stored `renderedDescription` instead of + re-rendering. + + Assertions across the whole set: no describable-but-unlisted id; no entry + scope-checked against one origin while its handler dispatches to another; no + read-scoped token having a destructive tool's **call** admitted (the handler + re-derives the tier from its own captured upstream annotations); `generation` + increments exactly once per paused rebuild and not at all on a guarded no-op + reload; and no case where a stale definition leads to a call that *succeeds* + against the wrong schema. +- **Publication API** (D13 rule 1): `buildDirectModeTools` returns a non-nil + catalog on every path — including the `DiscoverTools` error path — and the + single publisher `RefreshDirectModeTools` (which the D15 initial rebuild calls + rather than duplicating) does `SetTools` strictly before + `publishDirectCatalog`. Asserted by a test that fails if the catalog pointer is + swapped from inside the builder. +- **Event subscription ordering** (R14): a `servers.changed` published + immediately after `NewServer` returns still reaches the direct rebuild — the + regression test for hoisting `SubscribeEvents()` ahead of + `StartBackgroundInitialization`. +- **Initial registration** (D15/R14): on a freshly initialized proxy with **zero** + upstream servers, `p.directServer` already lists `describe_tool` before any + `servers.changed` fires; and an unrelated `config.reloaded` on that proxy + triggers no `SetTools` and no `tools/list_changed` (the FR-014 no-churn rule, + which a nil catalog would otherwise defeat). +- **Check-mode projection** (D14): a direct check response is byte-compatible with + the retrieve-surface response for the same underlying tool state — same + `status`/`reason`/`action`/`retryable` values, `tool_`/`server_` prefixes intact, + `tool_blocked_by_user` still distinct from `tool_denied_by_config`. +- Validation: missing-required → `invalid_params` with embedded full schema + hint + in both modes; validates against stored schema (stale full-mode client scenario); + fail-open on uncompilable schema; non-argument failures unchanged (FR-012/013). +- Config: validation messages (incl. `schema_deferred` rejection), env/flag + precedence, `DetectConfigChanges` clause. +- Rebuild guard: serialization flip → rebuild + notification; unrelated config edit + → no `SetTools` call (assert no notification) (FR-014); a flip while + `DiscoverTools` is failing (empty catalog published, mode recorded) still + rebuilds and is not lost (research.md R8). + +**Token gates (SC-001/SC-002)**: measure deferred vs full direct `tools/list` over +the frozen 45-tool corpus with the spec-083 profiler's pinned tokenizer (the +085/099 pattern): assert ≥70% reduction and ≥80% one-shot-callable (non-lossy) +share. Concretely this is a **new arm** in `bench/arms/` (registry contract: +`specs/083-discovery-profiler/contracts/arm-interface.md`, `bench/arms/arm.go`), +alongside `baseline`/`compact_sig`/`toon`/`tron`/`tscg`, plus its +`bench/arms/testdata/*_golden.txt` render golden — the existing arms all encode +`retrieve_tools` result sets, none renders a direct `tools/list`. `bench/` is +therefore an in-scope directory for this feature (added to the touch list below). + +**E2E** (`internal/server/e2e_test.go`): live flip with a connected client — +`notifications/tools/list_changed` observed, next listing reflects the mode, +tool set identical (SC-006); guessed-wrong → one self-healing retry succeeds +(SC-003); legacy aliases `/v1/tool_code`/`/v1/tool-code` covered by the same +mode/notification assertions as `/mcp` (FR-003). Local runs use the CI skip regex +for `internal/server` and an isolated high-port instance; never +`scripts/test-api-e2e.sh` against a shared machine. + +## Documentation & wiring checklist (Constitution VI) + +- `make swagger` after the config struct change (generated artifacts committed). +- `docs/configuration.md`: `direct_tool_response_mode` + env + flag. +- New feature doc under `docs/features/` (deferral convention, describe_tool on + direct, client-compat notes: schema-driven form UIs render empty forms; stale + cached listings both directions safe; instructions delta on `initialize`). +- `CLAUDE.md`: MCP-protocol built-ins line (describe_tool availability) + Recent + Changes entry. +- Issue linkage: commits use `Related #971` (never `Fixes`/`Closes`). + +## Complexity Tracking + +| Violation | Why Needed | Simpler Alternative Rejected Because | +|-----------|------------|-------------------------------------| +| `atomic.Pointer` swap for `directCatalog` (Principle II prefers channel-owned actors) | The catalog is written only by the single routing-refresh listener goroutine and read on every direct `tools/list`, `describe_tool`, and dispatch; FR-017 requires the listing, resolver, and validator to observe one consistent snapshot. | A channel-owned actor to serve reads of an immutable snapshot adds a goroutine + request/response channels around data that never mutates after publication. An atomic pointer to an immutable value is the idiomatic Go read-mostly pattern, is strictly less code, and is the same trade Spec 085's constitution check already accepted for the signature cache. The existing mutex-guarded `directToolPermissions` map it replaces was already shared-memory; this change reduces lock scope to zero on the read path. | + +| **FR-017's literal "rebuilt atomically, so no window exposes one without the other" is delivered as a safety property, not as a transaction** (spec.md FR-017 third bullet) | The advertised entry lives in mcp-go's tool registry and the catalog lives in ours; they are two publications. mcp-go reads its own `s.tools` under `toolsMu` before invoking our tool filters, so no lock of ours can span the registry read, and the only free-form field on `mcp.Tool` (`Meta`) marshals to `_meta`, so a generation stamp would move bytes FR-015 freezes. | Making it literally atomic would require either forking/patching mcp-go's registry or moving the direct listing off `SetTools` onto a hand-rolled tools/list handler — a far larger blast radius than this feature, and one that would put mcpproxy's own code on the protocol hot path for every routing mode. Instead R13 states the property that actually matters and proves it per-window: *no request observes a state less restrictive than both generations, and no request receives a definition for a name the registry is not currently serving*. The half of FR-017 that names dispatch — the handler validating against what it advertised — IS literally satisfied (closure capture, R9). **This narrowing is deliberate and needs the maintainer's assent at the tasks stage; it is the one place this plan does not deliver a spec MUST word-for-word.** | + +| **A schema-only change — input OR output — can be described one generation stale for the duration of the two-publication window** (the second FR-017 narrowing; it also narrows SC-007) | The skew discriminator is the rendered description, which does not encode either schema: full mode renders `[server] description` (`mcp_routing.go:95`) and deferred mode's appended signature is lossy by design, so a nested-property edit can render identically. No field on the registered `mcp.Tool` encodes the schema in *both* modes — deferred advertises the same `{"type":"object"}` placeholder for every tool — and `Tool.Meta` is wire-visible, so a hash stamp is closed off by FR-015. | The alternatives are the same ones rejected above (fork mcp-go's registry, or take the direct listing off `SetTools`). The residual is bounded and self-correcting by a mechanism this feature itself ships: dispatch is never wrong (R9 closure capture), so an agent acting on a stale **input** schema is rejected by the pre-dispatch validator with the **correct** schema embedded and succeeds on one retry — the US3/SC-003 path. Cost is one extra round trip, in a microsecond window, for a tool whose schema changed mid-refresh. A stale **output** schema has no such self-healing path — nothing validates against it — but also no safety consequence: output schemas are advisory shape metadata that MCP permits to be absent entirely (R2), so at worst an agent parses one response against last generation's shape, and the next publish corrects it. **Like the row above, this needs the maintainer's assent at the tasks stage.** | + +| **An annotations-only change can be listed/described one generation stale for the duration of the same window** (the third narrowing — of FR-011/SC-005 as well as FR-017; cross-model review round 8) | The round-5 draft avoided this by deriving the permission tier from the *registered* `mcp.Tool.Annotations`. That is not implementable: `contracts.DeriveCallWith` takes `*config.ToolAnnotations`, and more importantly `mcp.NewTool` seeds `destructiveHint=true` on every tool and each `With…HintAnnotation` overwrites only its own field, so the registered annotations say "destructive" for essentially every tool — which would hide almost the whole catalog from read- and write-scoped tokens and disagree with dispatch. The tier must therefore come from the catalog entry's upstream-derived `requiredPermission`, which is one publication behind for the width of the window. | Stamping the tier on the registered entry needs a free field on `mcp.Tool`; the only one (`Meta`) is wire-visible and FR-015 freezes those bytes — the same wall the row above hits. The residual is bounded by the mechanism that actually matters: **authorization at call time never uses the catalog.** `makeDirectModeHandler` re-derives the tier from the upstream annotations its own registration captured (`mcp_routing.go:211-213`), so a tool that just became destructive cannot be *called* by a read-scoped token even while the listing is briefly stale. The exposure is a listing/describe decision, in a microsecond window, self-correcting at the next publish. **Needs the maintainer's assent at the tasks stage, with the two rows above.** | + +No other deviations: no new dependency, no new package, no new abstraction beyond +the catalog type the spec's FR-017 mandates by name. diff --git a/specs/102-schema-deferred/quickstart.md b/specs/102-schema-deferred/quickstart.md new file mode 100644 index 000000000..ec569c798 --- /dev/null +++ b/specs/102-schema-deferred/quickstart.md @@ -0,0 +1,100 @@ +# Quickstart: Schema-Deferred Direct Mode + +Build, enable, and verify the deferred direct surface locally. Use an isolated +instance (high port + scratch `--data-dir` and `--config`) — never a shared core. + +## 1. Build + +```bash +go build -o mcpproxy ./cmd/mcpproxy +go build -tags server -o /dev/null ./cmd/mcpproxy # server edition still compiles +``` + +## 2. Enable deferral (opt-in) + +`~/.mcpproxy-dev/config.json` (scratch dir): + +```json +{ + "listen": "127.0.0.1:18102", + "direct_tool_response_mode": "deferred", + "mcpServers": [ { "name": "everything", "command": "npx", "args": ["-y", "@modelcontextprotocol/server-everything"], "protocol": "stdio", "enabled": true } ] +} +``` + +Equivalents: `MCPPROXY_DIRECT_TOOL_RESPONSE_MODE=deferred` or +`--direct-tool-response-mode=deferred`. Default (`full`) keeps pre-feature bytes. + +```bash +./mcpproxy serve --config ~/.mcpproxy-dev/config.json --data-dir ~/.mcpproxy-dev/data +``` + +## 3. Inspect the deferred listing + +```bash +# /mcp/all is a Streamable HTTP endpoint: a bare tools/list POST answers +# "Invalid session ID". Do the handshake first and reuse the session id. +BASE=http://127.0.0.1:18102/mcp/all +ACCEPT='Accept: application/json, text/event-stream' + +SID=$(curl -s -D- -o /dev/null "$BASE" -H 'Content-Type: application/json' -H "$ACCEPT" \ + -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"quickstart","version":"1"}}}' \ + | awk -F': ' 'tolower($1)=="mcp-session-id"{print $2}' | tr -d '\r') + +curl -s "$BASE" -H 'Content-Type: application/json' -H "$ACCEPT" -H "Mcp-Session-Id: $SID" \ + -d '{"jsonrpc":"2.0","method":"notifications/initialized"}' > /dev/null + +# Responses come back as SSE frames, so strip the "data: " prefix before jq. +curl -s "$BASE" -H 'Content-Type: application/json' -H "$ACCEPT" -H "Mcp-Session-Id: $SID" \ + -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \ + | sed -n 's/^data: //p' | jq '.result.tools[] | {name, inputSchema, description}' +``` + +Expect: every tool listed; `inputSchema == {"type":"object"}` on every **upstream** +entry — the `describe_tool` built-in keeps its real parameter schema in both +modes, since deferral governs upstream serialization only; `describe_tool` +present in the list; the `initialize` result carrying the convention +instructions. Upstream descriptions end with a `(param*:type, …)` signature +**for every tool the signature cache has warmed** — a tool the indexer has not +reached yet (or whose hash was just evicted) is listed with no suffix, which is +correct, not a bug: FR-005 lists a cache miss signature-less rather than dropping +or delaying it. Re-run the listing after indexing settles and the suffix appears. + +## 4. Recover a schema and trigger self-healing + +```bash +# Full schema for a lossy tool (both id forms work): +... method tools/call, name describe_tool, args {"tool_ids":["everything__sampleLLM"]} + +# Wrong-guess call → invalid_params error embedding the full input schema + hint: +... method tools/call, name everything__echo, args {} +``` + +## 5. Verify hot-reload flip (FR-014) + +Keep an MCP session open on `/mcp/all`, edit `direct_tool_response_mode` in the +config file, and observe `notifications/tools/list_changed` followed by the next +`tools/list` reflecting the new mode with an identical tool set. An unrelated +config edit must produce no notification. + +## 6. Test gates + +```bash +go test -race ./internal/toolsig/... ./internal/config/... ./internal/runtime/... +# internal/server locally: use the CI skip regex (unit-tests.yml) — see memory note. +/opt/homebrew/bin/golangci-lint run --config .github/.golangci.yml ./... +``` + +Golden rules: only `toolslist_goldens/{default_server,retrieve_tools_mode}.json` +are regenerated (once, FR-009 prose); `pre099/`, `code_execution_mode.json`, +`tools_list_prefeature.golden.json`, **both** `retrieve_full_default.golden.json` +and `retrieve_full_stats.golden.json` (one test pins both), and +`describe_plain_corpus/pre099.json` must pass **unregenerated** — the +describe-corpus prose movement is absorbed by adding named substitutions to +`describePlainDelta`, not by rewriting its golden. The new +`direct_mode_builtins.json` gate pins the direct built-ins in both modes and is a +standalone test, not an entry in `toolsListGoldenSurfaces`. + +Expected red tests during implementation (enumerated, not regressions): +`TestDescribeTool_RegisteredInRetrieveToolsModeOnly`'s `direct routing mode` +subtest, which asserts the v1 decision this feature reverses. diff --git a/specs/102-schema-deferred/research.md b/specs/102-schema-deferred/research.md new file mode 100644 index 000000000..3a6616d2d --- /dev/null +++ b/specs/102-schema-deferred/research.md @@ -0,0 +1,956 @@ +# Phase 0 Research: Schema-Deferred Direct Mode + +**Spec**: [spec.md](spec.md) · **Plan**: [plan.md](plan.md) + +Resolves the three `[NEEDS CLARIFICATION]` markers in the spec (R1–R3), records the +channel decision FR-007 demands (R4), and pins the mechanical findings the plan's +design rests on (R5–R13). Every code reference was verified against this tree +(base: `origin/main`, post-#1026). R11–R13 and the D9–D14 decisions were added +during self-review and two cross-model review rounds; each states the claim it +corrects. + +--- + +## R1 — Config surface (FR-001): dedicated key, not the shared axis + +**Decision**: New dedicated field `direct_tool_response_mode: "full" | "deferred"`, +default `"full"`, beside `tool_response_mode` in `internal/config/config.go`. Full +axis parity: serve flag `--direct-tool-response-mode` (help: "direct-surface +tools/list serialization: full (default) or deferred") and env alias +`MCPPROXY_DIRECT_TOOL_RESPONSE_MODE`, following the exact Spec-085 wiring pattern +(`cmd/mcpproxy/main.go:141` + `applyToolResponseModeFlag`, `loader.go:717`). + +**Rationale**: +- Extending the existing `tool_response_mode: "compact"` to also govern the direct + surface would silently change `/mcp/all` output for every deployment already + running compact — a violation of FR-015's byte-stability promise and of the Spec + 085 discipline that a serialization flip is an explicit operator act per surface. +- The existing flag/env pair is documented as "retrieve_tools serialization mode" + (`main.go:141`); the one-axis resolution would mutate the meaning of a shipped + flag and env var out from under scripts that set them. +- The two axes also have different value sets (`compact` vs `deferred`): the direct + surface's deferred rendering is not the compact retrieve_tools entry shape (it + keeps full descriptions and annotations, appends the signature, and substitutes a + placeholder schema), so sharing one enum would conflate two serializations. +- Cost: one more config field, wired through all four silent points (see R7). + +**Alternatives considered**: (a) one-axis `tool_response_mode` extension — rejected +above; (b) per-endpoint setting (`/mcp/all` vs `/mcp` direct) — rejected by FR-003 +(no per-endpoint divergence); (c) value `schema_deferred` on `routing_mode` — +rejected by the maintainer's accepted direction and FR-002 (the validator gains a +targeted error message naming the composition instead). + +## R2 — Output schemas (FR-006): defer them too; describe_tool carries them + +**Decision**: Deferred entries strip `outputSchema` along with `inputSchema` +(today applied by `applyToolOutputSchemaJSON`, `mcp_routing.go:143`). To keep the +information reachable, `describe_tool` definitions gain an additive +`output_schema` field — present only when the resolved tool declares one — on +**all** surfaces, emitted by the describe_tool definition assembly in +`mcp_describe_tool.go` (after `buildToolEntry`), **not** by `buildFullToolEntry`. + +**Rationale**: +- Output schemas are part of the ~77% schema share of the payload (Spec 083); the + MCP protocol allows structured content without a declared schema, so stripping is + protocol-safe, and a permissive placeholder is not even needed (`outputSchema` is + optional — the deferred entry simply omits it). +- Placement matters for the frozen goldens: `ToolMetadata.OutputSchemaJSON` is + already indexed (`internal/index/bleve.go:39,177`), so describe_tool can render + it from either resolver. Adding it inside `buildFullToolEntry` would change + full-mode `retrieve_tools` responses and break the byte-exact + `TestRetrieveToolsFullMode_GoldenByteIdentity` golden + (`retrieve_full_default.golden.json`) — so the field is added at the definition + assembly seam only. The definition-equality tests ("full-mode entry minus score") + are updated to enumerate the new field. +- **Correction — a response-bytes golden DOES exist.** An earlier draft of this + decision claimed "no golden pins describe_tool *response* bytes". That is + false: `TestDescribeToolPlainCorpus_ByteIdenticalWithOneEnumeratedDelta` + (`describe_plain_corpus_test.go`, + golden `testdata/describe_plain_corpus/pre099.json`) replays 18 plain-mode + `describe_tool` calls and compares each **response body byte for byte**, + permitting only the substitutions enumerated in `describePlainDelta`; its doc + comment is explicit that "a reordered key, a reworded remediation, a changed + cap message" fails. `output_schema` is `omitempty` and today's + `seedVisibilityFixture` tools declare no output schema, so D2 by itself does + not move these bytes — but the R5 prose work does, and any fixture that later + declares an output schema would. This gate is enumerated as the FOURTH frozen + gate in plan §Test strategy. +- Uniform across surfaces: a per-surface response shape would break the + single-assembly invariant and give the same id two different answers. + +**Alternatives considered**: keep `outputSchema` on deferred entries (wastes the +tokens the feature exists to save; Atlassian's low-compression precedent strips +it); direct-surface-only `output_schema` (surface divergence, rejected). + +## R3 — describe_tool batch cap (FR-009): keep 5 + +**Decision**: The definition-mode cap stays `maxDescribeToolIDs = 5` on the direct +surface, identical to the existing surfaces. `check: true` keeps its 50-id cap. + +**Rationale**: One builder feeds every registration (`buildDescribeToolTool`, +`mcp_describe_tool.go:62`) and its prose/cap are pinned by the tools/list goldens; +a per-surface cap would force a second builder or a parameterized schema — new +drift surface for ~zero benefit. The bulk-dump loophole the cap closes is *more* +live on a surface that enumerates the whole catalog, not less. An agent planning N +lossy tools batches ⌈N/5⌉ calls; SC-002's ≥80% non-lossy share bounds how often +that happens, and check-mode (cap 50) already covers the "gate a whole plan" +case without shipping schemas. + +## R4 — In-band convention channel (FR-007): direct-server instructions + +**Decision**: The deferral convention is carried by **MCP server instructions on +the direct server instance** — a static string set via +`mcpserver.WithInstructions(...)` when `p.directServer` is constructed in +`initRoutingModeServers` (`mcp_routing.go:619`), present in BOTH serialization +modes, phrased conditionally so it is true in both. + +The single reference wording lives in +[contracts/direct-deferred-surface.md §2](contracts/direct-deferred-surface.md) +— it is not restated here, so the two documents cannot drift. Exact bytes are +finalized at implementation and pinned by the new direct-surface built-in gate +(plan §Test strategy). + +**Rationale**: +- The `describe_tool`-description channel is budget-capped + (`describeToolTokenBudget = 250`, currently measuring 243) and golden-pinned; it + cannot absorb the signature legend, and cramming the *direct surface's* listing + convention into a tool that also serves two retrieve_tools surfaces would make + its prose wrong somewhere (the exact FR-009 trap). +- Instructions arrive in `initialize`, i.e. at Catalog time before `tools/list` — + the only channel the agent is guaranteed to see before reading the deferred + entries. +- Static-in-both-modes (FR-007's first branch) rather than emitted-only-when- + deferred: mcp-go instructions are constructor-time state; emitting them + conditionally would require rebuilding the server instance on hot-reload while + sessions span the flip. The conditional phrasing keeps them accurate when + deferral is off. +- Client-visible delta: the direct server's `initialize` response gains an + `instructions` field where it had none (today only the default retrieve_tools + server carries instructions, `mcp.go:480`). Enumerated as a back-compat note in + the plan and pinned by a test. + +**Precedence against the existing `instructions` config field** (interaction the +first draft missed): `instructions` is an operator-configurable config key +(`config.go:489`) that `resolveInstructions` (`mcp.go:94`) resolves to the +operator's string, or `defaultInstructions` when empty — and it is applied to +`p.server` ONLY (`mcp.go:480`). A hard-coded direct-server string would leave an +operator's configured instructions permanently unreachable on `/mcp/all` and on +`/mcp` under `routing_mode: "direct"`. **Decision**: the direct server's +instructions are the operator's `cfg.Instructions` when non-empty, **otherwise a +direct-specific default** — NOT `resolveInstructions`'s built-in +`defaultInstructions`, whose DISCOVERY line reads "Use 'retrieve_tools' to search +for tools by description" and whose CALLING line names `call_tool_read/write/ +destructive` (`mcp.go:72-80`). None of those tools exist on the direct surface +(`buildDirectModeTools` is upstream tools plus `describe_tool`), so importing +that text would put the in-band guidance FR-007 exists to provide in direct +contradiction with the listing, pointing agents at a nonexistent tool. The +direct-specific default keeps only what this surface actually exposes — the +`server__tool` naming, `describe_tool`, and the ABOUT links — and the deferral +legend follows after a blank line. It must NOT mention `upstream_servers` +either: management tools are registered by the code-exec and call-tool builders +only (`mcp_routing.go:398`, `:507`), never by `buildDirectModeTools`, so naming +it would repeat the very defect this decision fixes. One helper, +`resolveDirectInstructions(custom string)`, is the single place this composition +lives, so `resolveInstructions` keeps its current callers and bytes untouched. +Operator text, when set, still leads and is never rewritten. +The legend is a package constant so the new direct built-in golden can pin its +exact bytes, and the golden fixture uses the empty-`instructions` default so the +pinned string is deterministic. A test asserts a custom `instructions` value +still appears, legend still appended. Caveat, stated rather than implied: +instructions are constructor-time state on both server instances, so a change to +the `instructions` key still needs a restart — this feature does not make it +hot-reloadable and does not regress it either (the default server behaves the +same today). Only `direct_tool_response_mode` is hot-reloadable (FR-014), and the +legend is phrased to stay true across a flip precisely so no instructions rebuild +is required. + +**Alternatives considered**: describe_tool description (rejected above); synthetic +pseudo-tool (forbidden by FR-007); per-entry legend repetition (token-multiplied +across every tool — defeats the feature). + +## R5 — FR-009 prose: surface-neutral rewrite + one deliberate golden regen + +**Decision**: Keep ONE `buildDescribeToolTool` builder; rewrite its +retrieve_tools-specific prose to surface-neutral prose, keeping the marshalled +definition under the 250-token budget (`TestDescribeTool_DefinitionTokenBudget`, +currently measuring 243 — the rewrite has ~7 tokens of headroom and MUST be +re-measured, not assumed). + +**Scope — FOUR strings, not one.** The top-level description is only the most +visible offender; the same builder and handler carry three more +retrieve_tools-specific strings that are equally false on a surface with no +`retrieve_tools`, and two of them are *runtime response* strings rather than +definition bytes: + +| Site | Current text | Why it must change | +|---|---|---| +| `mcp_describe_tool.go:64` tool description | "…for specific tools found via retrieve_tools." | FR-009 (definition bytes; golden-pinned) | +| `mcp_describe_tool.go:71` `tool_ids` param description | "Tool ids in '\:\' format from retrieve_tools. Max 5, or 50 with check:true." | definition bytes; ALSO contradicts FR-011, which requires the direct surface to accept `server__tool` ids — the prose must name both accepted forms | +| `mcp_describe_tool.go:82` `filters` param description | "check:true only. Annotation filters, as in retrieve_tools." | definition bytes; golden-pinned | +| `mcp_describe_tool.go:50` `describeNotFoundRemediation` + `:195` malformed-id remediation | "…re-run retrieve_tools." / "…exactly as returned by retrieve_tools." | **response** bytes — contract §3 calls these the direct surface's "standard remediation" / "format remediation"; pinned by the describe_plain_corpus gate (R2 correction) | + +**Golden handling**: +- Regenerate `toolslist_goldens/default_server.json` and + `toolslist_goldens/retrieve_tools_mode.json` **once**, as the FR-010-sanctioned + enumerated delta. `toolsListAllowedDelta` needs **no edit**: it already lists + `describe_tool` for both surfaces (`toolslist_snapshot_test.go:188-192`), + because spec 099 enumerated it there. The frozen `pre099/` baseline and + `code_execution_mode.json` are never touched. +- Extend `describePlainDelta` (`describe_plain_corpus_test.go:48`) with the named + remediation substitutions, in the same one-substitution-per-scenario style the + existing entries use, and regenerate nothing there — the delta map is the + enumeration, the golden stays frozen. + +**Rationale**: The alternative — per-surface descriptions — breaks the deliberate +single-builder invariant ("the two schemas cannot drift", `mcp_describe_tool.go:53`) +and would need its own drift guard; the spec explicitly blesses the one-time regen +path (FR-009/FR-010), and Spec 099 already established the exact +regenerate-deliberately-once + enumerated-delta procedure in +`toolslist_snapshot_test.go`. + +## R6 — Signature cache read path (FR-005): add a non-compiling `Peek` + +**Finding**: `toolsig.Cache` exposes only `Get(hash, paramsJSON, description)` +(compiles + memoizes on miss, `cache.go:44`) and `Warm` (index-time). FR-005 +requires a lookup that (a) never compiles on the request path and (b) makes a miss +observable so the entry can be listed signature-less. + +**Decision**: Add `Peek(hash string) (Signature, bool)` to `internal/toolsig` — +pure read, no memoization, miss returns `ok=false`. Existing callers +(`buildCompactToolEntry`, warm path) are untouched. + +**Key availability confirmed**: the catalog's `hash` is real on the discovery +projection — `core/client.go:384` sets `toolMeta.Hash = +hash.ComputeToolHashWithOutputSchema(...)` inside `ListTools`, so every +`DiscoverTools` entry carries the same Spec-032 hash the indexer later warms the +cache with (`runtime/lifecycle.go:874`, which skips hashless tools). `Peek` and +the warm path therefore agree by construction; no hash is recomputed in the +direct rebuild. + +## R13 — Two publications, one generation: closing the catalog/registry skew window + +**Finding** (cross-model review, verified): the catalog pointer swap and +`SetTools` are two separate publications — `RefreshDirectModeTools` builds the +tools (publishing the catalog) and *then* calls +`p.directServer.SetTools(...)` (`mcp_routing.go:666-673`). An `atomic.Pointer` +makes the catalog swap atomic; it does **not** make catalog-plus-mcp-go-registry +a single transaction, and mcp-go's registry read happens inside its own +`ListTools` path where no lock of ours can span it. So the first draft's claim +that entry and handler "are rebuilt atomically and can never diverge" is only +half true, and FR-017's "no window exposes one without the other" is not +satisfied by the pointer alone. + +**What IS already atomic**: handler ↔ its own schema/hash. The handler closes +over its `directCatalogEntry`, so a dispatch can never validate against a +definition other than the one its own registration advertised, regardless of +skew. That half of FR-017 needs no further mechanism. + +**What skews**: the request-time *filters* and the *describe resolver*, which +read the published pointer independently of which registry generation produced +the list they are filtering. Enumerating both orderings: + +| Window | catalog published first | `SetTools` first (**chosen**) | +|---|---|---| +| Tool removed in the new generation | listing still shows it (old registry) while the filter finds no catalog entry — a *leak* unless the filter denies | already out of the registry: not listed, and rule 4's `GetTool` check makes describe answer `not_found`. **Window-free.** | +| Tool added in the new generation | not listed; describe would resolve from the new catalog → describable-but-unlisted | listed, but the old catalog has no entry → rule 2 denies it for filtered sessions and rule 4 answers `not_found`. **Fails closed.** | +| Same name, definition changed | filter/describe run ahead of the registry | registry runs ahead of the catalog: describe briefly returns the *previous* definition — which is the one the caller's own listing advertised. Accepted; see below. | + +`SetTools` first wins on every row: removal becomes window-free and addition +fails closed, whereas catalog-first makes removal a potential leak. The two +publications sit adjacent in the same function with nothing between them. + +**What cannot be built, stated plainly.** There is no way to stamp the generation +onto the registered entry and compare it at read time: `mcp.Tool`'s only +free-form field is `Meta`, which marshals to `_meta` on the wire +(`mcp/tools.go` `Tool.Meta`), so using it would move the very bytes FR-015 +freezes. And mcp-go takes its own snapshot of `s.tools` under `toolsMu` *before* +invoking our tool filters, so no lock of ours can span the registry read either. +The plan therefore does not promise a single transaction; it promises a +**safety property** and enumerates the residual: + +> No request observes a state less restrictive than **both** generations, and no +> request receives a definition for a name the registry is not currently serving. + +**Decision** — five rules that deliver that property: + +1. **`SetTools` runs first, the catalog is published immediately after**, in the + same function with nothing between them. Note what this forbids: the *builder* + must not publish. `buildDirectCatalog` returns the snapshot, + `renderDirectTools` turns it into `[]ServerTool`, and only + `RefreshDirectModeTools` — the single publisher, which the D15 initial rebuild + calls rather than duplicating — publishes, after its `SetTools` call. Today's `buildDirectModeTools` does the opposite, calling + `setDirectToolPermissions` mid-build (`mcp_routing.go:82`, `:151`) before + `RefreshDirectModeTools` reaches `SetTools` (`:673`); that ordering is exactly + what this rule reverses. This is the ordering that makes + *removal* window-free (a removed name leaves the registry before the catalog + stops describing it, and rule **4**'s registry check then answers + `not_found`), and it makes an *addition* fail closed (listed but not yet in + the catalog → rule 2 denies it for filtered sessions, rule **4** answers + `not_found`). +2. **Filters deny on catalog miss.** A name absent from the current catalog is + dropped, NOT passed through. Pass-through is reserved for built-ins and is + matched against an explicit built-in name set (today: `describe_tool`), never + inferred from "not in the catalog" — otherwise a stale upstream entry in the + old registry would skip the permission-tier gate entirely, turning the skew + window into a scope leak. (This corrects the fall-through wording in D10.) + **One explicit exception**: a *nil* catalog pointer — never published yet — + is not a miss, it is "no catalog", and the filters keep today's + `ParseDirectToolName` behavior for it. Otherwise a pre-init request, and every + hand-constructed test proxy that calls the filters directly + (`mcp_routing_test.go`, `toon_surface_isolation_test.go`), would see the whole + listing denied. An *empty published* catalog is a real generation and does + deny; only the nil pointer falls back. +3. **The permission tier comes from the catalog entry's `requiredPermission`, + which is derived from the UPSTREAM annotations — never from the registered + `mcp.Tool.Annotations`.** *(Corrected in cross-model review round 8; the + round-5 draft said the opposite, and it was wrong twice over.)* + + The draft rule read the tier off the entry the filter is filtering, via + `contracts.DeriveCallWith(tool.Annotations)`, "exactly as dispatch does". Two + verified defects: + + - **Type.** `DeriveCallWith` takes `*config.ToolAnnotations` + (`internal/contracts/intent.go:208`). The registered entry carries + `mcp.ToolAnnotation` — a different type. Dispatch calls it on the + **upstream** `*config.ToolAnnotations` the handler closed over + (`mcp_routing.go:211-213`), not on anything registered. + - **Semantics, and this one is severe.** `mcp.NewTool` seeds every tool with + `ReadOnlyHint=false, DestructiveHint=true, IdempotentHint=false, + OpenWorldHint=true` (`mcp/tools.go` `NewTool`), and each + `With…HintAnnotation` option overwrites only its own field. So a read-only + upstream tool registers with `readOnlyHint=true` **and** the untouched + default `destructiveHint=true`. `DeriveCallWith` gives destructive top + priority, so reading the tier off the registered entry would classify + nearly every direct tool as **destructive**, hiding it from every read- and + write-scoped agent token — and diverging from dispatch, which would still + admit the call as `read`. That is a listing↔dispatch divergence, precisely + the class of defect this design exists to remove. + + **Rule as it now stands**: the tier is the catalog entry's + `requiredPermission` (`requiredPermissionForDirectTool(upstreamAnnotations)`, + `mcp_direct_scope.go:18` — byte-identical to today's behavior and to + dispatch's), used **only after the rule-5 discriminator has confirmed the + catalog entry and the registered entry are the same generation**. The + discriminator, not the annotation source, is what buys generation-correctness + here. + + **Residual this creates, stated exactly**: an **annotations-only** change + whose rendered description is byte-identical passes the discriminator, so for + the duration of the window the listing and describe may gate on the previous + generation's tier. It is bounded the same way the schema-only residual is: + **dispatch is never wrong** — `makeDirectModeHandler` re-derives the tier from + the annotations *its own* registration captured (`mcp_routing.go:211-213`), so + a call is never mis-authorized, only a listing can be one generation stale. + This is the third and last item on the residual list below; it needs the same + maintainer assent as the other two. The catalog is also still the source for + the `(server, tool)` split (rule 2) — that is what `ParseDirectToolName` gets + wrong. +4. **Describe requires registry membership as well as catalog visibility.** + Before returning a definition the direct resolver checks + `p.directServer.GetTool(displayName) != nil` (mcp-go + `server/server.go:1023`, an O(1) read of the very map the listing is served + from). Catalog visibility ∧ registry presence closes describable-but-unlisted + in *both* orderings, and it strengthens rather than weakens FR-017's + "membership is decided by the direct-surface snapshot, not index presence" — + the registry is the listing's own source, not the index. The describe + resolver's permission tier follows rule 3 exactly — the catalog entry's + `requiredPermission`, after the rule-5 discriminator — for the same reason + rule 3 gives: the registered entry's `mcp.ToolAnnotation` carries mcp-go's + constructor defaults and cannot be used as a tier source. `GetTool` is + consulted here **only** for registry membership, never for annotations. +5. **A display name never denotes two origins — within a generation, or across + the window.** Two mechanisms, because the hazard has two shapes: + - *Within* a generation: display-name collisions (`a` + `b__c` vs `a__b` + `c`) + are **withheld — neither entry is registered** — with a warning naming both + pairs, replacing the first-writer-wins guard the earlier draft proposed. The + spec sanctions this explicitly ("MUST resolve a colliding id deterministically + **or report it** rather than guess", Edge Cases), and it is strictly safer + than today's behavior, which is not merely non-deterministic but *undefined*: + `buildDirectModeTools` appends both entries and `SetTools` collapses them + last-writer-wins into a map whose input order comes from a map iteration over + `m.clients`. Cost: two genuinely colliding tools become unreachable via the + direct surface until an operator renames one — a documented, logged, + pathological case, and the canonical `server:tool` route is unaffected. + - *Across* the window: a name can still change origin without ever colliding — + server A removed and server B added in one reconcile, where B's tool flattens + to A's old display name. Neither generation has a collision, yet the old + catalog's `(server, tool)` would be used to scope-check the new registered + entry. Closed by a **generation discriminator**: both the listing filters and + the describe resolver compare the entry's **stored** `renderedDescription` + — the exact string `renderDirectTools` handed to `SetTools` for that entry, + captured on the entry at render time — against the registered + `mcp.Tool.Description`, and **fail closed** (deny / `not_found`) on a + mismatch. One string comparison, no wire change. + + It must be the stored value, never a re-render. The deferred suffix comes + from `toolsig.Cache`, which mutates on its own schedule — `Warm` adds + entries at index time and `RetainHashes` evicts them + (`internal/toolsig/cache.go:79`, `:108`), neither of which triggers a direct + rebuild. Recomputing the description at filter or describe time would + therefore flip after a cache miss→warm (or hit→eviction) with no generation + change at all, manufacturing a false mismatch that leaves a still-registered + tool unlisted and undescribable **until the next rebuild** — a persistent + "listed ⇒ describable" violation (SC-007), far worse than the microsecond + window the discriminator exists to police. Comparing two immutable snapshots + has no such failure mode. + + **What the discriminator does NOT catch** (cross-model review round 6, + correcting an over-broad claim in the round-5 draft that it "subsumes" the + schema-change case): a **schema-only** change leaves the description + identical. The direct description is `fmt.Sprintf("[%s] %s", ServerName, + Description)` (`mcp_routing.go:95`) in full mode, and in deferred mode the + appended signature is lossy by design, so a nested-property edit can render + the same `~`. No field on the registered `mcp.Tool` reliably encodes the + schema in *both* modes either — deferred advertises the same + `{"type":"object"}` placeholder for every tool — and `Tool.Meta` is + wire-visible, so a hash stamp is closed off by FR-015 (see the top of this + section). The identity/origin hazards ARE fully closed; the schema-only one is + not, and it is recorded as a residual rather than papered over. + +**Residual, stated exactly** (rounds 5, 6, 8 and 9). Four different things happen +in the window, and three of them are still open: + +- *Anything the discriminator sees* — an origin flip, a description change, a + signature change — is **withheld**: the filters drop it and describe answers + `not_found`, rather than being answered from the stale generation. That is + availability loss, not correctness loss: fail-closed, self-correcting on the + next instruction, and bounded from the client's side by the + `tools/list_changed` notification `SetTools` has already emitted. +**What is invisible to the discriminator, field by field** (round 9, corrected in +rounds 10 and 11). + +What the renderer actually reads, stated exactly: in **full** mode +`fmt.Sprintf("[%s] %s", serverName, description)` (`mcp_routing.go:95`); in +**deferred** mode that same string plus `"\n" + toolName + +toolsig.Render(paramsJSON, description).Sig` when `Peek(hash)` hits, and nothing +appended when it misses. So `renderedDescription` is a function of `serverName`, +`toolName`, `description`, `paramsJSON`, the serialization mode, and whether the +signature cache held `hash` at render time — **not** of `description` alone. + +**Scope of this walk**: it covers the per-entry *definition* fields, because the +discriminator is a per-name comparison. The catalog-level fields are generation +state, not definition data, and are accounted for separately: `entries`, +`byDisplayName` and `byCanonical` express **membership**, whose changes are the +add / remove / origin-flip / collision cases already in the test matrix — a name +that leaves or joins them is caught by rule 2 (deny on catalog miss) and rule 4 +(registry membership), not by the description comparison; `serializationMode` is +*visible* (a flip adds or removes the whole signature suffix, which is why the +FR-014 guard rebuilds on it); and `generation` is never rendered and is +observability only. None of them is an invisible per-entry skew axis. + +Now the entry itself, split into **source** fields (set from the upstream +projection) and **derived** fields, because only the source fields are +independent axes of change: + +*Source fields.* + +| Field | Visible in a rendered-description comparison? | +|---|---| +| `serverName`, `toolName` | Yes. `serverName` is in the `[server]` prefix in both modes; `toolName` is in the deferred suffix. `displayName` is derived from the pair (`serverName__toolName`) and cannot move on its own. | +| `description` | Yes, by construction — it *is* the rendered string in full mode. | +| `paramsJSON` | **No, not reliably.** Never rendered in full mode. In deferred mode a semantic edit moves the hash, so it usually shows up — either as a different `Sig` or, until the new hash is warmed, as a vanished suffix. But an edit confined to nested properties the 085 grammar collapses to `~` renders a byte-identical `Sig` once that hash is warmed, and is then invisible. | +| `outputSchemaJSON` | **No.** Never rendered in either mode. It moves the Spec-032 hash, so it can flip a `Peek` hit to a miss and change the suffix that way — but only until the indexer warms the new hash, after which `Render` reproduces the identical `Sig` (it never reads the output schema). | +| `annotations` | **No.** Never rendered in either mode. | + +*Derived fields — no new axis.* The property that matters is one-directional: +each **cannot move unless one of its sources moves**, so none of them adds a +fourth independent case. The converse does not hold, and the plan does not claim +it. + +| Field | Derived from | Note | +|---|---|---| +| `renderedDescription` | the renderer inputs listed above, at render time | it *is* the discriminator | +| `hash` | server, tool, description, input schema, output schema (`hash.ToolHashWithOutputSchema`, `internal/hash/hash.go:10-16`) — **not** annotations | schemas are parsed and re-marshalled before hashing (`:59-104`), so a purely representational edit is not a schema change at all: same hash, same signature, and nothing for the validator to behave differently about. A *semantic* schema change always moves the hash, which is why the input-schema skew case needs its new hash warmed before the rebuild renders (see the test recipe in plan §Test Strategy). | +| `requiredPermission` | `annotations`, via `requiredPermissionForDirectTool` (`mcp_direct_scope.go:18`) → `contracts.DeriveCallWith` | it reads only `destructiveHint` and `readOnlyHint` (`internal/contracts/intent.go:208-227`), so a title / idempotent / open-world edit moves `annotations` without moving the tier. It is the *mechanism* of the annotations residual, not a fourth one. | + +So there are exactly **three independent source fields** that can change while the +rendered description stays byte-identical — `paramsJSON`, `outputSchemaJSON`, +`annotations` — and each is enumerated as a residual below. The skew tests drive +those three classes (the annotations case using a `read → destructive` edit, +since that is the sub-change the tier is sensitive to) and assert the derived +consequences. + +- *A schema-only change* (`paramsJSON`) passes the discriminator, so for those few + instructions `describe_tool` may return the **previous** generation's + `inputSchema` for a name whose registered handler already carries the new one. + (Note that in deferred mode a `paramsJSON` edit usually *is* visible, because it + moves the Spec-032 hash and the rendered signature with it — but not reliably: + a re-ordered or semantically-equal edit can render the identical signature, and + full mode never renders the schema at all.) Its blast radius is small + and, importantly, is absorbed by a mechanism this very feature ships: dispatch + is never wrong (the handler validates against the schema it captured, R9), so + an agent that acted on the stale definition is rejected by the pre-dispatch + validator with the **correct** schema embedded (FR-012/FR-013) and succeeds on + one retry — precisely the US3/SC-003 self-healing path. Cost: one extra round + trip, in a microsecond window, for a tool whose schema changed mid-refresh. +- *An output-schema-only change* (`outputSchemaJSON`) is invisible for the same + reason, and — unlike the input schema — **the pre-dispatch validator does not + absorb it**: nothing validates against an output schema, so there is no + self-healing retry here. Stated plainly rather than folded into the row above + (round 9): for the width of the window `describe_tool` may return the previous + generation's `output_schema` for a tool that now declares a different one. The + consequence is bounded by what an output schema *is* on this protocol — + advisory shape metadata for structured content, which MCP permits without any + declared schema at all (R2). A stale one cannot cause a call to be + mis-authorized, mis-validated, or rejected; at worst an agent parses the + response against last generation's shape. It self-corrects at the catalog + publish microseconds later, and a client that re-reads after the + `tools/list_changed` notification never sees it. +- *An annotations-only change* whose rendered description is byte-identical also + passes the discriminator (round 8, with rule 3's correction), so the listing + filters and the describe resolver may gate on the **previous** generation's + permission tier for those few instructions. Bounded identically: the tier that + actually authorizes a call is re-derived inside `makeDirectModeHandler` from + the annotations *its own* registration captured (`mcp_routing.go:211-213`), so + no call is ever mis-authorized — only a listing/describe decision can be one + generation stale, and it self-corrects at the catalog publish that follows + microseconds later. + +What is explicitly NOT residual any more: no session can be scope-checked against +one origin and dispatched to another; no display name can denote two origins, in +any interleaving; and no *call* can be authorized against a tier the registered +handler does not itself derive. + +**These three residuals narrow FR-017's "no window exposes one without the +other", and the annotations-only one additionally narrows FR-011/SC-005 while the +two schema ones narrow FR-017/SC-007. All are recorded in plan §Complexity +Tracking and cross-referenced from spec.md FR-017 for the maintainer's assent at +the tasks stage**, alongside the transaction narrowing — they are not to be +discovered later in a task. + +The catalog carries a monotonically increasing `generation`. It is not +decorative: every publish logs it beside the entry count and the serialization +mode, and the skew tests below assert on it (each paused rebuild must show +exactly one increment, and a guarded no-op reload must show none), which is what +makes "did this request see the old or the new generation" observable rather than +inferred. + +**Tests** — a rebuild paused between the two publications, with a concurrent +scoped `tools/list` and `describe_tool`, covering **eight** cases: an **added** +name, a **removed** name, a **same-name description-visible** change, a +**same-name origin flip** (server A removed and server B added in one reconcile, +B's tool flattening to A's old display name), a **within-generation collision** +(both entries withheld, warning logged, neither listed nor describable, in both +generations), and the three cases the discriminator cannot see: an +**input-schema-only change**, an **output-schema-only change**, and an +**annotations-only change** (read→destructive), each with the description and +rendered signature held byte-identical. Those three are exactly the *independent +source* fields that can move invisibly (the field table above; the derived +`hash` and `requiredPermission` add no fourth case), so the set doubles as the +proof of that enumeration. They assert the documented residuals behave as +claimed — the stale definition/tier may be returned, while dispatch still +validates against the new input schema and re-derives the new tier, so the +`invalid_params` error carries the NEW schema (one retry succeeds) and the +destructive call is still refused a read-scoped token at the handler; the +output-schema case asserts only that a stale advisory `output_schema` may be +returned and is corrected by the next publish. Plus the **two no-rebuild +cases** of rule 5: a signature-cache +**miss→warm** and a **hit→eviction** between registration and a later +filter/describe call (`toolsig.Cache.Warm` / `RetainHashes`), both of which must +leave the tool listed and describable — the proof that the discriminator reads +the stored `renderedDescription` rather than re-rendering. + +Assertions across the set: no describable-but-unlisted id; no entry scope-checked +against one origin while its handler dispatches to another; no read-scoped token +having a destructive tool's **call** admitted; and no case where a stale +definition leads to a call that *succeeds* against the wrong schema. + +## R14 — The direct surface is never initialized eagerly (FR-009/FR-014 gap) + +**Finding** (cross-model review round 3, verified): `initRoutingModeServers` +registers the built-ins for the code-exec and call-tool servers but deliberately +registers **nothing** on the direct server: + +> `// Note: Direct mode tools are built lazily/on-demand via RefreshDirectModeTools` +> `// because upstream servers may not be connected yet during initialization.` +> `// The servers.changed event will trigger a refresh.` +> (`mcp_routing.go:651-653`) + +Composing `describe_tool` into `buildDirectModeTools` (FR-018) is therefore +necessary but **not sufficient**: until something calls +`RefreshDirectModeTools`, the direct server has an empty tool map and FR-009's +"present in BOTH serialization modes" is false. The exposure is worst in exactly +the configuration the new built-in golden models — **zero upstream servers**, +where a `servers.changed` refresh may never be a meaningful trigger — and the +golden itself does not catch it, because it asserts over +`buildDirectModeTools()`'s return value rather than over what `p.directServer` +actually serves. + +It also interacts with the R8 nil-catalog rule: with no initial rebuild the +catalog stays nil, so "a nil catalog rebuilds unconditionally" turns the first +unrelated `config.reloaded` into a rebuild plus a `tools/list_changed` broadcast +— precisely the churn FR-014's no-op requirement forbids. + +**Decision**: +1. `initRoutingModeServers` performs one **initial direct rebuild** by calling + `RefreshDirectModeTools()` — the single publisher (R12), not a second copy of + the build→`SetTools`→publish sequence — before the HTTP listeners are wired, + so no session can ever observe an empty direct surface. With no upstreams connected yet this seeds exactly the built-in set + (`describe_tool`) and an empty catalog recording the effective mode; the + `servers.changed` refresh then replaces it normally. The stale comment at + `:651-653` is updated rather than left contradicting the code. +2. Because the catalog is now always published after init, the nil-catalog + "rebuild unconditionally" branch becomes unreachable in production and is kept + only as the defensive path (R8, R13 rule 2's nil exception). FR-014's no-op + guarantee then holds from the first reload onward. + + **This removes an accidental self-heal, so one ordering bug must be fixed with + it** (cross-model review round 8, verified): the routing-refresh listener + creates its subscription *inside* its own goroutine — + `eventCh := s.runtime.SubscribeEvents()` is the first line of + `listenForRoutingModeRefresh` (`server.go:528-530`) — and that goroutine is + merely *scheduled* at `server.go:301`, one line before + `s.runtime.StartBackgroundInitialization()` (`:302`). Nothing orders the + subscription before the first publish, and the bus delivers only to + subscribers already registered, so a fast background init can drop the first + `servers.changed` outright. Today that is self-healing by accident: the + catalog is nil, so the next `config.reloaded` rebuilds unconditionally. With + D15 publishing an empty catalog at init and FR-014 guarding the reload on a + mode comparison, the unconditional rebuild no longer fires and a dropped first + event would leave the direct surface at built-ins-only until the next upstream + change. **Decision**: `SubscribeEvents()` is hoisted into the constructor — + called synchronously before `StartBackgroundInitialization()`, with the + channel handed to the goroutine — so the subscription provably precedes any + publish. `server.go` is already in the touch list for the FR-014 guard; this + is the same file. A test asserts a `servers.changed` published immediately + after construction still reaches the direct rebuild. +3. The new direct built-in gate asserts over **`p.directServer.ListTools()`** — + what the server actually serves after init — not over + `buildDirectModeTools()`'s return value, so it fails if the initial + registration is ever dropped again. A companion test asserts an unrelated + `config.reloaded` on a freshly initialized zero-upstream proxy triggers no + `SetTools` and no notification. + +## R11 — The `{"type":"object"}` placeholder needs `RawInputSchema` (verified empirically) + +**Finding** (probe run against mcp-go v0.57.0 in this tree): the plan's first +draft asserted that "`mcp.NewTool` already produces" the permissive placeholder. +It does not. `ToolInputSchema.MarshalJSON` → +`toolArgumentsSchemaMarshalJSON` (`mcp/tools.go:765-792`) **unconditionally** +writes `properties` (as `{}` when nil) and `required` (as `[]` when empty), so a +schemaless `mcp.NewTool` entry serializes as: + +```json +"inputSchema":{"properties":{},"required":[],"type":"object"} +``` + +That is not the FR-004 wire shape, and it is not cosmetic: an empty declared +`properties` map is exactly the arg-pruning hazard FR-004's "never literal `{}`" +rule exists to avoid — a client that prunes arguments to the declared properties +would drop **every** argument. + +**Decision**: deferred entries are built with +`mcp.NewToolWithRawSchema(displayName, description, json.RawMessage(`{"type":"object"}`))` +(`mcp/tools.go:877`), which marshals to exactly `"inputSchema":{"type":"object"}`. +Two traps the implementation MUST honor, both verified: + +1. `NewToolWithRawSchema` takes **no** `ToolOption`s and leaves `Annotations` + **zero-valued** — every hint pointer `nil`. Copying only the upstream hints + onto it is NOT enough (cross-model review round 8): `mcp.NewTool` seeds a tool + with `ReadOnlyHint=false, DestructiveHint=true, IdempotentHint=false, + OpenWorldHint=true` before any option runs, and each `With…HintAnnotation` + overwrites only its own field, so a full-mode entry whose upstream declares + only `readOnlyHint:true` still marshals **all four** hints. A deferred entry + built by copying just that one hint would marshal only `readOnlyHint`, and an + entry with nil upstream annotations would marshal none at all — different + wire bytes for the same tool in the two modes, violating FR-004 ("unchanged + annotations") and FR-008 (set identity includes annotations). + **Decision**: the deferred renderer seeds the returned `mcp.Tool.Annotations` + with mcp-go's exact `NewTool` defaults and *then* applies the same five + upstream overrides `buildDirectModeTools` applies today + (`mcp_routing.go:99-115`). The unit matrix asserts the marshalled + `annotations` object is byte-identical between the two modes for three + fixtures: nil upstream annotations, partial (one hint), and full. +2. Setting `RawInputSchema` on a tool that already went through `mcp.NewTool` + is a **hard marshal failure**, not a silent override: `Tool.MarshalJSON` + (`mcp/tools.go:677-680`) returns `errToolSchemaConflict` when both + `InputSchema.Type` and `RawInputSchema` are set, which would break the whole + `tools/list` response. Full-mode entries keep the `NewTool` path unchanged + (FR-015); only the deferred branch uses the raw-schema constructor. + +A unit test asserts the exact marshalled bytes of a deferred entry's +`inputSchema` and that a deferred entry marshals without error. + +## R12 — Direct-catalog test seam (`upstream.Manager` is concrete) + +**Finding**: `MCPProxyServer.upstreamManager` is a concrete `*upstream.Manager` +(`mcp.go:121`), not an interface, and `buildDirectModeTools` calls +`p.upstreamManager.DiscoverTools` directly (`mcp_routing.go:80`). +`discoverTools` only returns tools from **connected** clients +(`upstream/manager.go:1160-1176`), so no unit test can produce a deterministic +multi-tool direct listing today. That blocks most of the plan's unit matrix +(deferred rendering, full↔deferred set identity, collision determinism, +describe/listing parity, the FR-015 fixture capture) — the one exception is the +new direct built-in golden, which is specified over **zero** upstream tools and +works with the existing `createTestMCPProxyServer` harness. + +**Decision**: split `buildDirectModeTools` into a thin I/O wrapper plus two pure +functions the tests drive directly: + +```go +func (p *MCPProxyServer) buildDirectCatalog(tools []*config.ToolMetadata, mode string) *directCatalog +func (p *MCPProxyServer) renderDirectTools(cat *directCatalog) []mcpserver.ServerTool +// renderDirectTools also records each entry's renderedDescription on the +// catalog before it is published, so the R13 rule 5 discriminator compares +// two immutable snapshots rather than re-rendering against a mutable cache. + +// DiscoverTools → buildDirectCatalog → renderDirectTools. Returns BOTH halves: +// the rendered tool set AND the catalog it was rendered from, unpublished. +func (p *MCPProxyServer) buildDirectModeTools() ([]mcpserver.ServerTool, *directCatalog) +``` + +**The return type must change, and this is load-bearing** (cross-model review +round 8): R13 rule 1 forbids the builder from publishing, so if +`buildDirectModeTools` still returned only `[]ServerTool` — as the round-7 draft +of this section said, "keeps its current signature and call sites" — the catalog +it built would be unreachable and `RefreshDirectModeTools` would have nothing to +publish after its `SetTools` call (`mcp_routing.go:666-673`). The two-value +return is what makes rule 1 expressible. + +**There is exactly ONE publisher, and this is pinned deliberately.** +`RefreshDirectModeTools` is it; the D15 initial rebuild does not re-implement the +sequence, it *calls* `RefreshDirectModeTools()`. Two independent +build→`SetTools`→publish sequences would be two places for rule 1's ordering to +drift, for no gain — the init case differs only in that `DiscoverTools` returns +nothing yet. So the blast radius of the signature change is two call sites of the +*builder*: `RefreshDirectModeTools` (`mcp_routing.go:666`) and one test +(`mcp_describe_tool_test.go:369`, already being edited for FR-009); and one new +call site of the *publisher*: `initRoutingModeServers`. + +The publisher reads: + +```go +tools, cat := p.buildDirectModeTools() +p.directServer.SetTools(tools...) // rule 1: registry first +p.publishDirectCatalog(cat) // …catalog immediately after +``` + +`buildDirectModeTools` returns a non-nil catalog on **every** path, including the +`DiscoverTools` error path (`mcp_routing.go:81-85`, which returns `nil` today): +there it returns the built-ins-only tool set and an **empty catalog recording the +effective mode**, so FR-014's guard always has a mode to compare against and +FR-018's built-in survives an upstream outage. + +The unit matrix feeds `buildDirectCatalog` a fixture `[]*config.ToolMetadata` +slice (including the `__`-in-server-name collision pair) and asserts over +`renderDirectTools` output. No interface extraction and no new dependency; +`buildDirectModeTools` keeps its name and its two call sites, and gains the +second return value the publication rule requires (above). + +## R7 — Config-field wiring points (verified against this tree) + +A new config field is silently inert unless all four are wired: +1. **Hot-reload detection**: `DetectConfigChanges` + (`internal/runtime/config_hotreload.go:44`) needs a + `direct_tool_response_mode` clause (the `tool_response_mode` clause is at + :157 — same pattern), else an apply of only this field reports "no changes". +2. **Contract regen**: `make swagger` after the config struct change (swaggo v2; + generated artifacts are committed). +3. **Env override**: `MCPPROXY_DIRECT_TOOL_RESPONSE_MODE` in + `internal/config/loader.go` beside the Spec-085 alias (:717), validated by + `cfg.Validate()` after overrides. +4. **Docs**: `docs/configuration.md` (+ the feature doc, plan §Documentation). + +Additionally FR-002: the `routing_mode` validation error gains a special case for +the literal `schema_deferred` naming the supported composition (`config.go:2229` +area is the analogous `tool_response_mode` block; `routing_mode` validation sits +nearby). + +## R8 — Rebuild + notification mechanics (FR-014, FR-018) + +**Findings** (mcp-go v0.57.0, the pinned version in go.mod): +- `MCPServer.SetTools` replaces the tool map wholesale and **automatically emits + `notifications/tools/list_changed` to all initialized sessions** when the tools + capability declares listChanged (it does: `WithToolCapabilities(true)`). So + FR-014's notification comes free with the rebuild; the "no-op on unrelated + edits" requirement means *guarding the rebuild call*, not suppressing a + notification. +- The `config.reloaded` branch of `listenForRoutingModeRefresh` + (`internal/server/server.go:554`) currently calls only + `reapplyScannerSecurityConfig()` and `RefreshPrompts()` — confirmed: no direct + rebuild, exactly as the spec asserts. The fix is one call in this branch to a + new guard method that compares the serialization mode the current direct tool + set was built with (recorded on the catalog snapshot, R9) against the live + effective mode (`p.currentConfig()`, the live snapshot per + `profile_resolver.go:40` — never the construction-time `p.config`), and calls + `RefreshDirectModeTools()` only on a real change. This runs on the same single + listener goroutine as the servers.changed rebuild, so no new reentrancy. +- `SetTools` panics on task-tool name collisions and last-writer-wins on + duplicate names — motivates the deterministic collision guard in R9. +- `applyStrictInputSchemaDefault` (`server/server.go:884`) is inert here + (mcpproxy does not set `WithStrictInputSchemaDefault`), so `SetTools` injects + no `additionalProperties: false`. Note this is *not* what produces the + `{"type":"object"}` wire shape — see R11: the placeholder requires + `NewToolWithRawSchema`, and note further that `applyStrictInputSchemaDefault` + early-returns on `len(tool.RawInputSchema) > 0`, so a raw-schema entry would + stay permissive even if that option were ever enabled. +- **Rebuild guard and the empty catalog**: `buildDirectModeTools` returns `nil` + and clears permissions when `DiscoverTools` errors (`mcp_routing.go:81-85`). + The catalog MUST still be published in that case (zero entries, recording the + mode it was built with), so the FR-014 guard always has a mode to compare + against; a nil/absent catalog is treated as "mode unknown" and rebuilds + unconditionally, so a flip during an upstream outage is never lost. +- FR-018: `RefreshDirectModeTools` → `SetTools(buildDirectModeTools()...)` — the + built list is 100% upstream-derived today (`mcp_routing.go:76`), so + `describe_tool` must be appended inside `buildDirectModeTools` (tool-set + construction), never registered once beside it. + +## R9 — FR-017 catalog authority: one snapshot, four consumers + +**Finding** (confirming the spec's divergence analysis): the direct listing is a +projection of `upstreamManager.DiscoverTools` (server-level filtering only), while +`describe_tool`'s existing resolver (`toolVisibleToSession`, +`mcp_visibility.go:51`) starts with `toolIndexed` — index-backed and filtered by +tool-level approval at index time — and the signature cache is warmed only from +indexed tools. A pending/changed tool is therefore listed but unindexed: no +schema, no signature, index-backed describe says not_found. Also, +`makeDirectModeHandler` closes over `(serverName, toolName)` but *not* over the +schema it advertised; validation resolved through the index could see a different +definition during refresh skew. + +**Decision**: introduce a `directCatalog` — an immutable per-rebuild snapshot +(entry: display name, `(server, tool)` pair, description, `ParamsJSON`, +`OutputSchemaJSON`, `Hash`, annotations, required permission) built inside the +direct rebuild from the same `DiscoverTools` result the listing renders from, +stored via atomic pointer swap on `MCPProxyServer`, absorbing today's +`directToolPermissions` map. Consumers: (1) listing rendering (both modes), (2) +signature `Peek` by entry hash, (3) direct-surface describe_tool resolution +(display-name map = the registration mapping; no re-parsing), (4) pre-dispatch +validation — the handler closure captures its own entry's `ParamsJSON`+`Hash` at +build time, so it validates against exactly what it advertised. Collisions +(`a`+`b__c` vs `a__b`+`c`): the catalog build iterates a deterministically sorted +tool list and **withholds every colliding display name** — neither pair is +registered — logging a warning that names both origins. (An earlier draft kept +the first writer, mirroring the F7 prompt guard; round 5 showed that still lets +one display name denote different origins across a rebuild. See R13 rule 5.) +Describe and dispatch therefore agree by construction, because an ambiguous name +is never served at all. + +**Scope of the atomicity this buys**: handler ↔ its own schema/hash, which is +what consumer (4) needs. It does NOT make the catalog swap and `SetTools` one +transaction — see [R13](#r13--two-publications-one-generation-closing-the-catalogregistry-skew-window) +for the filter/resolver skew window and the five rules that close it. + +## R10 — Direct-surface describe_tool visibility parity (FR-011) + +**Finding**: the direct listing's filters are `filterDirectModeToolsForAuth` +(`mcp_direct_scope.go:61` — **profile scope for every auth type**, then token +server scope, then the operation-permission tier via +`lookupDirectToolPermission`) and `filterDirectToolsForAgentCallability` +(`mcp_direct_callability.go:49` — agent sessions only), both registered on +`p.directServer` (`mcp_routing.go:616-618`); profile scope is additionally +re-checked at dispatch (`mcp_routing.go:191`). `toolVisibleToSession` +(`mcp_visibility.go:51`) checks scope + callability but **no permission tier**, +and answers existence-confirming reason codes +(`quarantined`/`pending_approval`/`changed`/`disabled`) — both divergences the +spec names are real in this tree. (The step helpers live in three files, not one: +`serverInScope` in `mcp_visibility.go:150`, `evaluateToolGate` in +`tool_gate.go:80`, `isToolCallable` in `mcp.go:6068`.) + +**Four parity leaks the first draft did not close.** "Listing parity" is not +achieved by building a correct resolver alone — the *listing* side, the +definition assembly, and two suggestion paths must all be brought onto the same +catalog: + +1. **The listing filters still re-parse the display name.** Both filters resolve + `(server, tool)` via `ParseDirectToolName` (first-`__` split; + `mcp_direct_scope.go:76`, `mcp_direct_callability.go:65`), and + `filterDirectModeToolsForAuth` looks the permission up by display name. For a + server named `a__b` with tool `c` the filter evaluates a *nonexistent* server + `a` while the catalog-backed describe resolver evaluates the real pair + `a__b`/`c` — so describe can return a definition for a tool the same session's + listing dropped, which is precisely the FR-011 violation this decision exists + to prevent. **Decision**: both filters resolve through the catalog's + `byDisplayName` map and **deny on a catalog miss**, so listing and describe + agree by construction. Built-ins keep passing through, but matched against an + explicit built-in name set (today `describe_tool`) — never inferred from + "absent from the catalog", which during the R13 skew window would let a stale + upstream entry skip the permission-tier gate entirely. This makes `mcp_direct_scope.go` and + `mcp_direct_callability.go` in-scope files, and retires + `lookupDirectToolPermission`'s separate map in favor of the catalog entry's + `requiredPermission` (FR-017). +2. **`check:true` suggestions (`did_you_mean`) come from a server-level corpus.** + The shared preflight evaluator builds them in + `visibleCorpus.candidates()` (`internal/preflight/evaluator.go:594-650`), + which filters only by token server scope and server policy + (found/quarantined/enabled) — **no operation-permission tier and no + tool-level callability/approval gate** — and they are surfaced to the caller + (`mcp_describe_check.go:294`, `did_you_mean`). Swapping the *id gate* to + catalog membership does not touch this corpus, so a read-scoped token's + `not_found` could name destructive tools absent from its own direct + `tools/list` (FR-011, SC-005). The root cause is structural, not incidental: + `preflight.Scope` is a **server-name set** (`NewScope(profileName, allowed)`, + built by `sessionPreflightScope` from `serverInScope`, + `preflight_glue.go:209-226`), so the evaluator has no way to express a + per-tool gate. **Decision — and the injection point exists**: + `preflight.EvalContext.Index` is an interface, supplied today as + `&preflightIndexReader{index: p.index, annotations: annotations}` + (`preflight_glue.go:103`, reader at `:284`). The direct surface passes a + `directCatalogIndexReader` backed by the catalog snapshot and filtered by the + listing-parity gates instead, so both the id resolution and the `did_you_mean` + corpus come from the same authority (FR-017) with no change to the evaluator + itself. Suppressing `did_you_mean` on this surface is the fallback only if + that reader turns out to be load-bearing for a verdict path. A test asserts a + read-scoped token receives no suggestion naming a destructive tool. + + **The same adapter must canonicalize the id, or check mode is broken for + every direct id** (cross-model review, verified): ids reach the evaluator + untouched — `normalizeDescribeCheckIDs` forwards them verbatim + (`mcp_describe_check.go:192-204`) and `RunPreflightForSession` hands them + straight to `preflight.Evaluate` (`preflight_glue.go:182-188`) — and the + evaluator accepts colon ids ONLY: `splitToolID` is a + `strings.SplitN(id, ":", 2)` (`evaluator.go:500-510`) and `evaluateOne` + turns a non-split id into `not_found` with `detailMalformedID` + (`evaluator.go:205-212`). So `github__create_issue` under `check:true` would + answer `not_found` today no matter what the id gate does. The direct + check-mode adapter therefore MUST, per id, in this order: (1) resolve the id + against the catalog by display name or canonical name; (2) apply the + listing-parity gates, answering plain `not_found` for anything invisible + **without** consulting the evaluator; (3) canonicalize the survivors to + `server:tool` before `Evaluate`; (4) restore the caller's original id string + and the requested ordering in both the response entries and the activity + record, so an agent that sent `server__tool` gets `server__tool` back. No + change to `internal/preflight/evaluator.go` is required under this design — + the adapter lives on the server side — which is why the evaluator is + deliberately absent from the file-touch list; if implementation finds a + verdict path that cannot be expressed through the injected `IndexReader`, + adding an evaluator seam becomes a scope change to record, not a silent edit. +3. **The definition's annotations are not read from the entry you pass in.** + `buildFullToolEntry` (`mcp_entry_builder.go`) uses `result.Tool` only for + name/description/inputSchema/server, and resolves annotations through + `p.lookupToolAnnotations` (`mcp.go:6344`), which reads the **StateView + snapshot** — not `result.Tool.Annotations`. `call_with` is then derived from + whatever that lookup returned, defaulting to `contracts.ToolVariantRead` when + it returns nil. So synthesizing a `*config.ToolMetadata` from the catalog and + handing it to `buildToolEntry` would still resolve annotations out-of-band, + re-introducing the exact index/StateView dependency FR-017 removes — and for + the listed-but-unindexed pending/changed tool that SC-007 targets, the + definition would come back with **no annotations and `call_with: "read"` for a + destructive tool**. That is worse than a missing field: it is a wrong safety + hint. **Decision**: the definition-assembly seam takes an optional + annotations override, supplied from the catalog entry on the direct surface + only; the retrieve_tools surfaces pass nothing and keep the StateView lookup + byte-identical (protecting the `retrieve_full_default` golden and the + describe_plain_corpus gate). A test asserts a listed pending destructive tool + describes with its real annotations and `call_with: "destructive"`. +4. **Definition-mode case-correction suggestions.** `suggestCanonicalToolID` + (`mcp_visibility.go:204`) is invoked on the definition-mode not-found path + (`mcp_describe_tool.go:202-206`) and gates its suggestion with + `toolVisibleToSession` — the index-backed, permission-tier-blind resolver. + **Decision**: the direct surface uses the catalog-backed resolver for that + gate too (or omits the suggestion), same rule and same test. + +**Decision**: a direct-surface resolver (`directToolVisibleToSession`) built from +the catalog + the same step helpers (`serverInScope`, `evaluateToolGate`, +`isToolCallable`) with the mode split of FR-011: invisible-to-this-session → +plain `not_found` in both modes; visible → definition-mode always renders the +snapshot-backed definition (even pending / changed / tool-level-disabled states, +which non-agent direct listings retain — **server-level** quarantined or disabled +servers are dropped by `DiscoverTools` before projection +(`upstream/manager.go:1128-1138`) and so are simply unlisted and `not_found` +here, contract §3 note); `check: true` delegates to the shared +spec-098/099 preflight evaluator for the informative verdict, with the id gate +swapped to catalog membership so a listed-but-unindexed id is never short-circuited to +`not_found`. Retrieve-surface semantics untouched: `handleDescribeTool` gains a +per-surface resolver seam (the registration passes it), and the existing surfaces +keep the index-backed resolver byte-for-byte. diff --git a/specs/102-schema-deferred/spec.md b/specs/102-schema-deferred/spec.md index 1944af6ca..1b740c9a1 100644 --- a/specs/102-schema-deferred/spec.md +++ b/specs/102-schema-deferred/spec.md @@ -132,7 +132,7 @@ An operator flips deferred serialization on (or off) in config with the proxy ru - **Operation-permission tier**: the direct listing hides a tool whose required operation type exceeds the agent token's permissions; the existing resolver checks server scope and callability but no permission tier. Without this gate a read-scoped token could obtain the full schema of a destructive tool absent from its own `tools/list`. - **Existence-confirming reason codes**: the existing resolver deliberately answers `quarantined` / `pending_approval` / `changed` / `disabled` — codes that confirm the tool exists. On the direct surface those states are precisely what the listing filters hide from an agent session, so returning them would leak exactly what FR-011 forbids. The rule is parity with *this session's* listing, and it splits by mode: - **Invisible to this session** (the listing omitted it, for any reason): plain `not_found`, in BOTH definition and `check` mode. Never a reason code, never any other existence signal. - - **Visible to this session** (the listing included it) — note that a non-agent session's direct listing deliberately retains tools that are pending, changed, quarantined or disabled, since the callability filter narrows only agent-token sessions: + - **Visible to this session** (the listing included it) — note that a non-agent session's direct listing deliberately retains tools that are pending, changed or **tool-level** disabled, since the callability filter narrows only agent-token sessions. **Server-level** quarantine and server-level disable are a different thing and never reach this surface at all: `DiscoverTools` skips those clients outright (`internal/upstream/manager.go` — "Skipping quarantined client for tool discovery"), so their tools are absent from the projection for every session, agent-token or not, and there is nothing for the visibility rule to retain: - *definition mode* MUST return the snapshot-backed definition. A listed tool is never undescribable (US2 scenario 5, FR-017, SC-007); withholding the schema of a tool the session can already see buys no confidentiality and breaks the deferral contract, since under deferral the listing no longer carries the schema itself. - *`check: true`* MUST return the informative availability verdict (`pending_approval`, `changed`, `quarantined`, `disabled`) rather than `ready` — that verdict is the entire purpose of check mode, which exists to gate a plan before its first call. - retrieve_tools-surface semantics are unchanged by all of the above. @@ -144,7 +144,22 @@ An operator flips deferred serialization on (or off) in config with the proxy ru - **FR-017** (catalog authority): the direct surface MUST have ONE authoritative catalog snapshot, and the listing, the appended signature, `describe_tool` resolution, and pre-dispatch validation MUST all read from it. Today they would not: the listing is rebuilt from live upstream discovery, while both the signature cache and `describe_tool`'s resolver are index-backed, and the index is filtered by tool-level approval where the listing is filtered only at the server level. That divergence is not hypothetical (see the catalog-divergence edge case) and produces three distinct failures — a listed tool that cannot be described, a listed tool that can never carry a signature, and validation resolving a different schema than the handler was registered with during refresh skew. Requirements: - membership is decided by the direct-surface snapshot, not by index presence: a tool absent from the index but present in the listing MUST still be describable (definition mode, per FR-011) and validatable, and a tool present in the index but absent from the listing MUST report `not_found`; - each registered direct entry MUST carry the schema (and hash) it was built from, so its handler validates against the same definition it advertised; - - the advertised entry and its handler MUST be rebuilt atomically, so no window exposes one without the other. + - the advertised entry and its handler MUST be rebuilt together, **delivered as a safety property rather than as a transaction** (amended 2026-08-26; see the note below). Concretely: no request may observe a state LESS RESTRICTIVE than both the outgoing and incoming generations, and no request may receive a definition for a name the registry is not currently serving. The half of this that IS literally atomic stays literal — a handler validates against the definition its own registration captured, because it closes over it. + + > **Amendment (ACCEPTED 2026-08-26 by the maintainer; T001/T002/T003).** The third bullet above is written as "delivered as" rather than as an atomicity claim, because the claim was not deliverable: the advertised entry lives in mcp-go's tool registry and the catalog lives in mcpproxy's, and mcp-go reads its own tool map under its own lock before invoking our filters, so the two publications cannot be made one transaction without forking mcp-go or taking the direct listing off `SetTools`. The half of this bullet that names dispatch — the handler validating against what it advertised — IS literally satisfied. The plan substitutes a proven safety property plus enumerated residuals, all recorded in [plan.md §Complexity Tracking](plan.md#complexity-tracking) and **awaiting the maintainer's assent at the tasks stage**. + > + > **The narrowing is not confined to FR-017.** Because the two publications are not one transaction, three other requirements are absolute in steady state but not inside the microsecond publication window, and the plan says so rather than letting the contradiction sit unstated: + > - **FR-011 / SC-005** (permission-tier parity, `not_found` for a destructive tool a read-scoped token cannot list): an **annotations-only** change — a tool that becomes destructive with a byte-identical rendered description — may be listed and described against the *previous* generation's tier for the width of the window. Call-time authorization is NOT affected: the handler re-derives the tier from the annotations its own registration captured, so the call is refused even while the listing is stale. + > - **FR-017 / SC-007** (a listed tool is always describable with its current definition): a **schema-only** or **output-schema-only** change is equally invisible to the discriminator, so `describe_tool` may return the previous generation's `inputSchema` / `output_schema` for the same window. + > + > **All three narrowings are ACCEPTED** — FR-017's atomicity as a safety property (T001), a schema-only change describable one generation stale (T002), and an annotations-only change listed one generation stale (T003). They are recorded here, in the requirement text, rather than left in the plan's appendix, so a later reader sees what the spec actually promises instead of discovering the gap in an implementation note. + > + > **What makes each acceptable is a compensating mechanism this feature itself ships, not tolerance for the residual:** + > - the **input-schema** case self-heals — the pre-dispatch validator rejects the stale guess with the CORRECT schema embedded, so one retry succeeds (US3/SC-003); + > - the **annotations** case cannot mis-authorize — call-time authorization re-derives the tier from the annotations the handler captured, never from the catalog, so a read-scoped token's call to a newly-destructive tool is still refused; + > - the **output-schema** case has no self-healing path and is accepted on its merits: output schemas are advisory shape metadata MCP permits to be absent entirely, so the worst outcome is one response parsed against last generation's shape. + > + > The window is one publication wide and self-corrects at the next publish. What is NOT accepted is any of these being invisible: the skew tests (D13) assert the partition itself, so a fourth residual cannot appear without a failing test. - **FR-018** (rebuild survival): the direct-surface tool set is replaced wholesale on every upstream refresh — the refresh path sets the complete tool list, and that list is built exclusively from upstream discovery with no built-ins in it today. `describe_tool` MUST therefore be composed into direct-surface tool-set *construction*, not registered once alongside it, or the next upstream change would silently drop it. A regression test MUST assert `describe_tool` is still listed after a refresh. **Rollout & client compatibility (US4)** diff --git a/specs/102-schema-deferred/tasks.md b/specs/102-schema-deferred/tasks.md new file mode 100644 index 000000000..f2be07fe5 --- /dev/null +++ b/specs/102-schema-deferred/tasks.md @@ -0,0 +1,254 @@ +# Tasks: Schema-Deferred Direct Mode — Full Enumeration Without Schemas + +**Input**: Design documents from `/specs/102-schema-deferred/` +**Prerequisites**: [plan.md](plan.md) (required), [spec.md](spec.md) (user stories), [research.md](research.md), [data-model.md](data-model.md), [contracts/direct-deferred-surface.md](contracts/direct-deferred-surface.md) + +**Tests**: Test tasks ARE included. Constitution V (Test-Driven Development) is a PASS gate in plan.md — *"Every behavior lands test-first"* — so each behavioral task is preceded by its failing test. + +**Organization**: By user story, in spec priority order. US1 and US2 are both P1 and both required for the feature to make sense (deferral without the recovery stage trades tokens for failed calls); US3 and US4 are P2. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: parallelizable — different file, no dependency on an incomplete task +- **[Story]**: US1–US4; Setup / Foundational / Polish phases carry no story label + +--- + +## ⚠️ Phase 0: Open approvals (each blocks its OWN dependents, not everything) + +plan.md's Complexity Tracking records **three narrowings of normative spec MUSTs**, each ending *"needs the maintainer's assent at the tasks stage"*. That stage is now. These are not implementation choices — each one changes what the spec promises, so the answer belongs in spec.md, not in the plan's appendix. + +**They do not all block everything.** An earlier draft made Phase 0 a blanket gate; that stalls work the approvals cannot affect. The real dependencies: + +| Approval | Blocks | Does NOT block | +|---|---|---| +| T001 (FR-017 atomicity) | catalog **publication** (T017–T020) | config axis, `Peek`, rendering | +| T002 (stale schema) | pre-dispatch validation semantics (Phase 5) and the skew residual tests (T072+) | everything else | +| T003 (stale annotations) | the permission-tier and annotations-override work (T024, T053) | everything else | +| T004 (fold D1/D2/D3 into spec.md) | **Phase 1** — the config key it names is what Phase 1 builds | — | + +So T004 gates Phase 1; T001–T003 gate their own dependents and are otherwise concurrent with it. + +- [x] T001 Obtain and record the maintainer's decision on narrowing 1: **FR-017's "rebuilt atomically, so no window exposes one without the other" ships as a safety property, not a transaction.** The catalog and mcp-go's tool registry are two publications and cannot be made one without forking mcp-go or taking the direct listing off `SetTools`. Record the accepted wording in `specs/102-schema-deferred/spec.md` FR-017 (amend the MUST, or add an explicit "delivered as" clause naming D13's five rules). +- [x] T002 Obtain and record the maintainer's decision on narrowing 2: **a schema-only change (input or output) can be described one generation stale for the width of the publication window** — also a narrowing of SC-007. Record in `specs/102-schema-deferred/spec.md` beside FR-017/SC-007, including that the input case self-heals via the pre-dispatch validator (US3) and the output case does not but is advisory only. +- [x] T003 Obtain and record the maintainer's decision on narrowing 3: **an annotations-only change can be listed/described one generation stale** — a narrowing of FR-011 and SC-005. Record in `specs/102-schema-deferred/spec.md`, including the compensating property that call-time authorization never reads the catalog (`makeDirectModeHandler` re-derives the tier from its own captured upstream annotations). +- [ ] T004 Fold the plan's D1, D2 and D3 resolutions back into `specs/102-schema-deferred/spec.md`, removing the `[NEEDS CLARIFICATION]` markers in FR-001 (dedicated `direct_tool_response_mode` key), FR-006 (deferred entries strip `outputSchema`; `describe_tool` definitions gain additive `output_schema`) and the FR-009 batch-cap question (cap stays 5; 50 for `check:true`). A spec carrying unresolved markers while its plan treats them as decided is the drift this task exists to close. + +**Checkpoint**: spec.md contains no `[NEEDS CLARIFICATION]` marker and no normative MUST that the plan silently narrows. T004 is done before Phase 1 starts; T001–T003 are done before their named dependents. + +--- + +## Phase 1: Setup (shared infrastructure) + +**Purpose**: the config axis and the one cache primitive everything else reads through. No behavior change on any surface yet. + +- [ ] T005 Add `DirectToolResponseMode` field and the `ToolResponseModeDeferred` constant beside the existing `ToolResponseMode` declarations in `internal/config/config.go`, defaulting to `"full"`. +- [ ] T006 Add validation for `direct_tool_response_mode` (accepted values `full`, `deferred`) beside the existing `tool_response_mode` validation block in `internal/config/config.go`, and extend `routing_mode` validation so `"schema_deferred"` is rejected with a message naming the supported composition (`routing_mode: "direct"` + `direct_tool_response_mode: "deferred"`) per FR-002. +- [ ] T007 Write failing test (before T005/T006) for config validation messages and default resolution in `internal/config/config_test.go`: default is `full`; `deferred` accepted; an unknown value names both accepted values; `routing_mode: "schema_deferred"` produces the composition-naming error. +- [ ] T008 [P] Add the `MCPPROXY_DIRECT_TOOL_RESPONSE_MODE` environment alias beside the existing `MCPPROXY_TOOL_RESPONSE_MODE` alias in `internal/config/loader.go`. +- [ ] T009 [P] Add the `--direct-tool-response-mode` serve flag in `cmd/mcpproxy/main.go`, following the existing `--tool-response-mode` flag pattern, with help text that says *direct* surface (the existing flag's help says "retrieve_tools serialization mode" and must stay that way — FR-001 parity requirement). +- [ ] T010 [P] Write failing test for env/flag/config precedence for the new axis in `internal/config/loader_test.go`, mirroring the existing `tool_response_mode` precedence test. +- [ ] T011 [P] Write failing test for `toolsig.Cache.Peek` in `internal/toolsig/cache_test.go`: a warmed hash returns `(sig, true)`; an unwarmed hash returns `(zero, false)` **and does not compile or memoize** — assert the miss leaves the cache size unchanged, since an accessor that silently compiled would violate FR-005's "no per-request compilation". +- [ ] T012 Add the non-compiling, miss-reporting `Peek(hash) (Signature, bool)` method to `internal/toolsig/cache.go`, leaving the existing compiling accessor and the index-time `Warm` path untouched for their current callers (FR-005). + +**Checkpoint**: config axis exists and validates; `Peek` exists. Nothing reads either yet; every existing test still passes. + +--- + +## Phase 2: Foundational (BLOCKING — the catalog) + +**Purpose**: the `directCatalog` snapshot (FR-017) that listing, describe, validation and the discovery filters all resolve through. **No user story can begin until this phase is complete** — US1 renders from it, US2 resolves through it, US3 validates against it, US4 rebuilds it. + +**⚠️ D13 rule 1 is structural, not stylistic**: the builder must not publish. `buildDirectModeTools` returns the catalog alongside the tool set so the single publisher can `SetTools` *first* and swap the catalog immediately after. + +- [ ] T013 Write failing test for the pure builder seam in `internal/server/mcp_direct_catalog_test.go`: driving `buildDirectCatalog` with a fixture `[]*config.ToolMetadata` produces one entry per tool carrying display name, `(server, tool)` pair, description, `ParamsJSON`, `OutputSchemaJSON`, Spec-032 `Hash`, annotations and `requiredPermission`. Fixture-driven because `upstream.Manager` is concrete and `DiscoverTools` only returns tools from connected clients (research.md R12). +- [ ] T014 Write failing test for display-name collision withholding in `internal/server/mcp_direct_catalog_test.go`: server `a` + tool `b__c` and server `a__b` + tool `c` flatten to the same `a__b__c`; assert **neither is listed** and **neither resolves through the catalog**, and that a warning names both origins — in both flattening directions (D6/D13 rule 5). Today's code is undefined here, not merely non-deterministic. The matching *"neither is describable"* assertion is deliberately NOT here: describe resolution does not exist until T045/T046, so bundling it would make this Phase 2 test un-greenable inside Phase 2. It lands as T046a. +- [ ] T015 Create `internal/server/mcp_direct_catalog.go` with the immutable `directCatalog` type, a sorted deterministic build, collision withholding with both origins logged, and a `generation` counter logged per publish (D13). +- [ ] T016 Add the `atomic.Pointer[directCatalog]` field to `MCPProxyServer` in `internal/server/mcp.go` and the `publishDirectCatalog` swap in `internal/server/mcp_direct_catalog.go`; retire the mutex-guarded `directToolPermissions` map the catalog absorbs. +- [ ] T017 Write failing test in `internal/server/mcp_direct_catalog_test.go` asserting the catalog pointer is **never** swapped from inside the builder, and that the publisher orders `SetTools` strictly before `publishDirectCatalog` (D13 rule 1). +- [ ] T018 Split `buildDirectModeTools` into the pure pair `buildDirectCatalog` + `renderDirectTools` in `internal/server/mcp_routing.go`, returning `([]ServerTool, *directCatalog)` — a **non-nil catalog on every return path**, including the `DiscoverTools` error path that returns `nil` early today (D12). +- [ ] T019 Update the two `RefreshDirectModeTools` / test call sites for the new builder signature in `internal/server/mcp_routing.go`, making `RefreshDirectModeTools` the single publisher. +- [ ] T019a Change `makeDirectModeHandler`'s signature in `internal/server/mcp_routing.go` so it takes and captures its own `*directCatalogEntry`, and have `renderDirectTools` pass each entry as it builds the handler. **This production task was missing entirely from an earlier draft** — T020 updated call sites for a signature change no task performed, and Phase 5's validation task consumed a captured entry nothing had captured. The closure capture is what makes R9's "a dispatch can never validate against a definition other than the one its own registration advertised" true, so it belongs here, in the foundational phase, not with the validator that later reads it. +- [ ] T019b Append the `describe_tool` registration inside direct tool-set **construction** in `internal/server/mcp_routing.go`, on **every** return path including the `DiscoverTools` error path (FR-018). **Moved out of Phase 4**: T024 asserts a freshly initialized proxy already lists `describe_tool`, so leaving the registration in the describe phase made Phase 2 depend on Phase 4 — a cycle through its own prerequisite. Registration is a one-line append with no resolver behind it; the describe *handler* work stays in Phase 4, where it belongs. +- [ ] T020 Update `makeDirectModeHandler` call sites for the T019a signature change in `internal/server/mcp_routing_test.go`, `internal/server/toon_surface_isolation_test.go`, `internal/server/profile_pin_enforcement_test.go` and `internal/server/mcp_direct_callability_test.go`. Not `[P]`: it follows T019a, and two of these files also move under T023. +- [ ] T021 Write failing test for deny-on-catalog-miss in `internal/server/mcp_direct_scope_test.go`: an unknown display name is denied by the filters, built-ins are allowed by explicit name set, and a **nil** catalog is not treated as a miss (D13 rule 2). +- [ ] T022 Rewrite `filterDirectModeToolsForAuth` in `internal/server/mcp_direct_scope.go` to resolve `(server, tool)` through the catalog instead of `ParseDirectToolName`, taking the permission tier from the entry's upstream-derived `requiredPermission` — **never** from the registered `mcp.Tool.Annotations`, which carry mcp-go's `destructiveHint=true` default (D10/D13 rule 3). Retire `lookupDirectToolPermission`'s separate map. +- [ ] T023 [P] Rewrite `filterDirectToolsForAgentCallability` in `internal/server/mcp_direct_callability.go` to use the same catalog resolution (D10). +- [ ] T024 Write failing test for the D15 initial rebuild in `internal/server/mcp_routing_test.go`: on a freshly initialized proxy with **zero** upstream servers, the catalog is non-nil and `p.directServer` already lists `describe_tool` before any `servers.changed` fires. Depends on T019b for the registration and, for the tier assertions it makes, on approval T003. +- [ ] T025 Make `initRoutingModeServers` perform the initial direct rebuild by **calling** `RefreshDirectModeTools()` (not a second copy of the ordering) in `internal/server/mcp_routing.go` (D15). +- [ ] T026 Write failing regression test in `internal/server/server_test.go`: a `servers.changed` published immediately after `NewServer` returns still reaches the direct rebuild (R14). +- [ ] T027 Hoist `SubscribeEvents()` out of the listener goroutine into the constructor, ahead of `StartBackgroundInitialization`, in `internal/server/server.go` — publishing an empty catalog at init retires the nil-catalog accidental self-heal, so a dropped first `servers.changed` no longer recovers (D15/R14). + +**Checkpoint**: one authoritative catalog exists, is published in the right order, and both discovery filters resolve through it. Full-mode direct listings are unchanged; no user-visible behavior has moved yet. + +--- + +## Phase 3: User Story 1 — Deferred enumeration (Priority: P1) 🎯 MVP + +**Goal**: with `direct_tool_response_mode: "deferred"`, every visible upstream tool is still listed — same names, count and annotations — but each entry carries its compact signature and a minimal permissive schema instead of the upstream `inputSchema`/`outputSchema`. + +**Independent test**: enable deferral on a fixture proxy with multiple upstreams, fetch `tools/list` on the direct surface, and assert every tool present, no upstream schema properties, signature-suffixed descriptions, and ≥70% token reduction against the frozen 45-tool corpus. + +- [ ] T028 [US1] Write failing tests for deferred entry rendering in `internal/server/mcp_routing_deferred_test.go`: the marshalled `inputSchema` is byte-exactly `{"type":"object"}` — asserted **on the JSON, not the Go struct**, since `mcp.NewTool` marshals `{"properties":{},"required":[],"type":"object"}` (R11/D9); no upstream properties or required list; the description is the existing `[server] …` text with the Spec-085 signature appended. +- [ ] T029 [US1] Write failing test for annotations parity in `internal/server/mcp_routing_deferred_test.go`: the marshalled `annotations` object is **byte-identical to full mode's** for three fixtures — nil upstream annotations, one hint set, all five set — proving the raw-schema constructor was seeded with mcp-go's `NewTool` defaults (D9). Also assert a deferred entry marshals without `errToolSchemaConflict`. +- [ ] T030 [US1] Write failing test for the signature cache miss path in `internal/server/mcp_routing_deferred_test.go`: a tool whose hash is not warmed is listed **without** a signature suffix — never dropped, never delayed (FR-005). +- [ ] T031 [US1] Implement the deferred branch of `renderDirectTools` in `internal/server/mcp_routing.go` using `mcp.NewToolWithRawSchema(…, json.RawMessage(`{"type":"object"}`))`, seeded with mcp-go's `NewTool` annotation defaults and *then* given the upstream overrides (D9), reading signatures via `toolsig.Cache.Peek`. Leave the full-mode `NewTool` path untouched (FR-015). +- [ ] T032 [US1] Write failing test for set identity across modes in `internal/server/mcp_routing_deferred_test.go`: same names, count, annotations and ordering source in full and deferred (FR-008), including under agent-token and profile filters (SC-005/FR-016). +- [ ] T033 [US1] Write failing test for FR-015 byte-stability in `internal/server/mcp_routing_deferred_test.go` using the **live stdio fixture** form (reuse the `preflight_e2e_test.go` harness + `testdata/preflight_fixture_server.js`), which runs unchanged at the merge-base and on this branch. Assert deferral-off rendering is byte-identical to pre-feature output modulo the appended `describe_tool` entry. Fall back to the same-tree differential form only if the fixture is judged too heavy — and say which was used in the test's doc comment. +- [ ] T034 [US1] Write failing test for the direct-server instructions in `internal/server/mcp_routing_test.go`: a custom `instructions` config value still appears on the direct server's `initialize`, with the deferral legend appended, in **both** modes (D11). +- [ ] T035 [US1] Add `resolveDirectInstructions(custom string)` in `internal/server/mcp_routing.go` — custom when non-empty, else a **direct-specific default**, then the legend — and attach it via `WithInstructions` on `directServer` in `initRoutingModeServers`. Deliberately **not** `resolveInstructions`, whose `defaultInstructions` advertises `retrieve_tools` and `call_tool_*`; the direct default names only `server__tool` calling, `describe_tool` and the ABOUT links — never `upstream_servers`, which this surface does not register (D16). Safe only because T019b registers `describe_tool` back in Phase 2: naming a tool the surface does not expose would be exactly the D16 mistake one level up. + +- [ ] T035a [US1] Run the SC-001/SC-002 token gates for this phase's own claim: US1's stated independent test asserts **≥70% payload reduction** on the frozen 45-tool corpus, and no other Phase 3 task measures it. Use the spec-083 profiler pipeline directly here; the reusable `bench/arms/` arm and its render golden stay T075/T076 in Polish. Without this, Phase 3 cannot be signed off against the criterion it declares. + +**Checkpoint**: US1 is independently demonstrable — flip the config, fetch `tools/list`, observe the token drop with the full tool set intact. + +--- + +## Phase 4: User Story 2 — describe_tool on the direct surface (Priority: P1) + +**Goal**: `describe_tool` is present on the direct surface in both modes, accepts both id forms, resolves through the registration mapping, and never discloses anything the same session's `tools/list` would not. + +**Independent test**: call `describe_tool` with a direct `server__tool` id and assert the definition is field-equal to the Spec-085 full rendering; assert an out-of-visibility tool returns a per-id `not_found`. + +- [ ] T036 [US2] Invert and rename the `"direct routing mode"` subtest of `TestDescribeTool_RegisteredInRetrieveToolsModeOnly` in `internal/server/mcp_describe_tool_test.go` to assert **presence** on the direct surface (FR-009/FR-018). It is green on `origin/main` today, so its failure during implementation is expected, not a regression. +- [ ] T037 [US2] Append the `describe_tool` registration inside direct tool-set **construction** in `internal/server/mcp_routing.go`, on **every** return path including the `DiscoverTools` error path, so a `SetTools` refresh or an upstream hiccup can never drop the built-in (FR-018). +- [ ] T038 [US2] Rewrite all **five** retrieve_tools-specific strings in `internal/server/mcp_describe_tool.go` to be surface-neutral (D5/R5): the tool description, both parameter descriptions, the `describeNotFoundRemediation` constant and the inline malformed-id remediation. The `tool_ids` prose must name **both** accepted id forms, since FR-011 requires `server__tool` here. Keep the single builder. +- [ ] T039 [US2] Re-measure `TestDescribeTool_DefinitionTokenBudget` (`internal/server/mcp_describe_tool_test.go`) after T038. The 250-token budget currently sits at ~243, so naming both id forms may overrun it. If it does: shorten the prose first; raise the budget only with the same deliberate, documented justification the 150 → 250 bump carried. +- [ ] T040 [US2] Regenerate exactly `default_server.json` and `retrieve_tools_mode.json` under `internal/server/testdata/toolslist_goldens/` via the documented `MCPPROXY_WRITE_TOOLSLIST_GOLDENS` flow — the one enumerated FR-010 exception. `code_execution_mode.json` and the frozen `pre099/` baseline are **never** regenerated; a diff in either means the change reached further than claimed. +- [ ] T041 [US2] Extend `describePlainDelta` in `internal/server/describe_plain_corpus_test.go` with the named remediation substitutions, keeping `testdata/describe_plain_corpus/pre099.json` frozen. Widen the delta value to an ordered slice of substitutions if one string needs more than one — and update the assertion site accordingly. +- [ ] T042 [US2] Write failing test for both id forms in `internal/server/mcp_describe_direct_test.go`: canonical `server:tool` and direct `server__tool` resolve to the same definition through the **registration mapping**, including a server name that itself contains `__` — never by re-parsing the display name. +- [ ] T043 [US2] Write failing test for the permission-tier gate in `internal/server/mcp_describe_direct_test.go`: a read-scoped token gets `not_found` for a destructive tool absent from its own listing (the disclosure hole the existing resolver leaves open). +- [ ] T044 [US2] Write failing test for the catalog-divergence case in `internal/server/mcp_describe_direct_test.go`: a listed pending/changed tool returns its **snapshot-backed definition** in definition mode and a `pending_approval` / `changed` verdict under `check: true` (SC-007, FR-011/FR-017) — the case where a naive index-backed resolver answers `not_found` and makes deferral strictly worse than full mode. +- [ ] T045 [US2] Create `internal/server/mcp_describe_direct.go`: the catalog-backed direct resolver with listing-parity gates (token server scope + operation-permission tier + profile + agent callability), plain `not_found` for invisible ids in both modes, snapshot-backed definitions for visible ids. +- [ ] T046 [US2] Add the per-surface resolver seam to the `describe_tool` handler in `internal/server/mcp_describe_tool.go` — existing surfaces keep the index-backed `toolVisibleToSession` **byte-identical**; the direct surface gets the T045 resolver. +- [ ] T046a [US2] Write failing test for the collision case's describe half in `internal/server/mcp_describe_direct_test.go`: a withheld `__` collision resolves in **neither** id form — the assertion split out of T014, which could not go green inside Phase 2 because no resolver existed yet. +- [ ] T046b [US2] Write failing test for `output_schema` in `internal/server/mcp_describe_direct_test.go`: present iff the tool declares one, absent otherwise. **Before T047**, not after — an earlier draft ordered this test after its implementation, which cannot fail first and so does not satisfy the constitution's test-first gate. +- [ ] T047 [US2] Add the additive `output_schema` field at the **definition-assembly seam** in `internal/server/mcp_describe_tool.go` (D2), not in `buildFullToolEntry`, so full-mode retrieve_tools bytes are untouched (R2). +- [ ] T048 [US2] Re-run gate 4 (`TestDescribeToolPlainCorpus_ByteIdenticalWithOneEnumeratedDelta`) after T047 rather than assuming it: `output_schema` is `omitempty` and today's fixture tools declare none, but that must be verified, not asserted from the plan. +- [ ] T049 [US2] Write failing test for the annotations-override case in `internal/server/mcp_describe_direct_test.go`: a listed pending/changed **destructive** tool describes with its real annotations and `call_with: "destructive"` — `buildFullToolEntry` otherwise reads annotations from the StateView and would silently downgrade the safety hint to `read` (D10). +- [ ] T050 [US2] Add the optional catalog-supplied annotations override at the definition-assembly seam in `internal/server/mcp_entry_builder.go`, unused on the retrieve path so `TestRetrieveToolsFullMode_GoldenByteIdentity` passes unregenerated on **both** its goldens. +- [ ] T051 [US2] Write failing test for the check-mode adapter in `internal/server/mcp_describe_direct_test.go`: a `server__tool` id under `check: true` returns its verdict **under the id the caller sent** (D14) — without the canonicalize-and-restore path every direct id answers `not_found`, since `preflight.Evaluate` accepts colon ids only. +- [ ] T052 [US2] Implement the check-mode adapter in `internal/server/mcp_describe_direct.go`: canonicalize `server__tool` → `server:tool` before `preflight.Evaluate`, gate invisibility **without** consulting the evaluator, project `status`/`reason`/`action`/`retryable` through untouched, and restore the caller's original id and ordering in both the response and the activity record (D14). +- [ ] T053 [US2] Add the id-gate seam for the direct surface in `internal/server/mcp_describe_check.go`, leaving verdict logic unchanged. +- [ ] T054 [US2] Create `internal/server/preflight_glue.go`'s `directCatalogIndexReader` and pass it as `preflight.EvalContext.Index` on the direct surface, so id resolution **and** the `did_you_mean` corpus share the catalog authority (D10/FR-017). +- [ ] T055 [US2] Write failing test for suggestion discipline in `internal/server/mcp_describe_direct_test.go`: a read-scoped token's `check: true` miss returns no `did_you_mean` entry naming a destructive tool absent from its own listing, and the definition-mode case-correction suggestion is gated by the same catalog-backed resolver (D10). +- [ ] T056 [US2] Add the resolver seam to `suggestCanonicalToolID` in `internal/server/mcp_visibility.go`, keeping the existing retrieve-surface resolvers byte-identical. +- [ ] T057 [US2] Write failing test for listing↔describe parity **in both directions** in `internal/server/mcp_describe_direct_test.go`: with a server whose name contains `__`, compare the session's rendered `tools/list` name set against the set of ids that resolve in definition mode, for an admin session, a read-scoped token, a write-scoped token and a profile-pinned session. No id may be describable-but-unlisted (disclosure) or listed-but-undescribable (SC-007). +- [ ] T058 [US2] Write failing test in `internal/server/mcp_describe_direct_test.go` for a server removed between `tools/list` and `describe_tool`: the stale id returns a per-id not-found **without failing the batch**. (The `output_schema` half moved to T046b, where it precedes its implementation.) + +**Checkpoint**: Catalog → Inspect → Execute is complete on one surface. US1 + US2 together are the shippable increment. + +--- + +## Phase 5: User Story 3 — Self-healing direct calls (Priority: P2) + +**Goal**: a wrong guess from a lossy signature costs exactly one bounded retry, not an opaque upstream error. + +**Independent test**: call a direct tool omitting a required parameter; assert the error embeds the full input schema and a hint, and that a transport-level failure attaches no schema. + +- [ ] T059 [US3] Write failing test for pre-dispatch validation in `internal/server/mcp_routing_validation_test.go`: a missing required argument yields `invalid_params` with the tool's **full** input schema and a one-line hint embedded, in **both** serialization modes (self-healing is mode-independent). +- [ ] T060 [US3] Write failing test in `internal/server/mcp_routing_validation_test.go` that validation runs against the **stored upstream schema**, never the advertised `{"type":"object"}` placeholder — the stale-full-mode-client scenario (US3 scenario 4). +- [ ] T061 [US3] Write failing test in `internal/server/mcp_routing_validation_test.go` for fail-open: an uncompilable or unsupported stored schema dispatches exactly as today, counted in logs, never blocking a call a schemaless proxy would have allowed (FR-013b). +- [ ] T062 [US3] Write failing test in `internal/server/mcp_routing_validation_test.go` that non-argument failures (upstream down, auth, timeout) attach **no** schema and keep their current shapes. +- [ ] T063 [US3] Write failing test in `internal/server/mcp_routing_validation_test.go` asserting that a validation failure emits the `emitActivityToolCallStarted` + `emitActivityToolCallCompleted("error", …)` pair, so the rejected call is visible in the availability funnel. +- [ ] T064 [US3] Add pre-dispatch validation **and** its activity emission in one change in `internal/server/mcp_routing.go`: call `p.inputValidator.validateArgs` against the captured catalog entry's `ParamsJSON` (captured by T019a), placed **immediately after** `directToolCallabilityBlockWithReason` and **before** `markSessionWorked`, render the existing `invalidParamsErrorResult` on failure, and emit the started/completed-error pair matching the Spec-085 `call_tool_*` path. The unconditional `emitActivityToolCallStarted` stays on the dispatch path only. + + **These are deliberately one task.** An earlier draft split validation from its activity emission; landing the first alone would start rejecting calls that appear nowhere in the funnel — creating precisely the observability blind spot issue #969 established this handler must not have, and calling it an intermediate state. + +**Checkpoint**: deferral's worst case is one retry. + +--- + +## Phase 6: User Story 4 — Operator opt-in, hot-reload, cache-safe rollout (Priority: P2) + +**Goal**: flipping the setting on a running proxy rebuilds the direct surface and notifies connected clients, with no restart and no churn on unrelated config edits. + +**Independent test**: with a client connected, flip the config value; assert a `notifications/tools/list_changed` is emitted, the next `tools/list` reflects the new serialization, and the tool set is identical across the flip. + +- [ ] T065 [P] [US4] Add the `direct_tool_response_mode` clause to `DetectConfigChanges` in `internal/runtime/config_hotreload.go`, beside the existing `tool_response_mode` clause. +- [ ] T066 [US4] Write failing test in `internal/runtime/config_hotreload_test.go` that a change to the new field is detected and an unrelated edit is not. +- [ ] T067 [US4] Write failing test for the rebuild guard in `internal/server/server_test.go`: a serialization flip triggers a rebuild **and** a notification; an unrelated config edit triggers **no** `SetTools` and **no** notification (the FR-014 no-churn rule, which a nil catalog would otherwise defeat). +- [ ] T068 [US4] Add the guarded direct rebuild to the `config.reloaded` branch of `listenForRoutingModeRefresh` in `internal/server/server.go`: compare the mode the current catalog was built with against the live effective mode read via `p.currentConfig()` — **never** construction-time `p.config` — and rebuild only on a real change. +- [ ] T069 [US4] Write failing test (alongside T067, same file, before T068) in `internal/server/server_test.go` that a flip while `DiscoverTools` is failing still rebuilds and is not lost: an empty catalog is published with the new mode recorded (R8). +- [ ] T070 [US4] **Requires Phase 5 complete.** Write the eight publication-skew interleaving tests in `internal/server/mcp_direct_skew_test.go` (D13), with a rebuild paused between `SetTools` and the catalog publish and a concurrent scoped `tools/list` + `describe_tool`, in three groups: (1) closed by design — added name, removed name, description-visible change, origin flip; (2) closed by withholding — a within-generation display-name collision; (3) the three documented residuals — input-schema-only, output-schema-only, annotations-only (read → destructive). The input-schema case **must** be semantically different but signature-identical (a nested-property edit the 085 grammar collapses to `~`) with the new hash warmed before the rebuild renders, or `Peek` misses and the description changes visibly instead; a canonicalization-equal edit will not do, since `hash` re-marshals the parsed schema. +- [ ] T071 [US4] Write the two **no-rebuild** cache tests in `internal/server/mcp_direct_skew_test.go`: a signature-cache miss→warm and a hit→eviction between registration and a later filter/describe call must both leave the tool listed and describable — proof that the discriminator reads the **stored** `renderedDescription` rather than re-rendering. +- [ ] T072 [US4] **Requires Phase 5 complete** — its self-healing assertions are Phase 5 behaviour. Assert across the whole skew set in `internal/server/mcp_direct_skew_test.go`: no describable-but-unlisted id; no entry scope-checked against one origin while its handler dispatches to another; no read-scoped token having a destructive tool's **call** admitted; `generation` increments exactly once per paused rebuild and not at all on a guarded no-op reload; and no stale definition leading to a call that *succeeds* against the wrong schema. + +**Checkpoint**: the feature is operable on a running proxy and the concurrency story is proven rather than asserted. + +--- + +## Phase 7: Polish & cross-cutting + +- [ ] T073 Add the standalone direct built-in gate to `internal/server/toolslist_snapshot_test.go` with `testdata/toolslist_goldens/direct_mode_builtins.json`: zero upstream tools (listing = `describe_tool` only), **both** serialization modes, membership + byte-exact serialization, plus the direct server's instructions string captured with an empty `instructions` config so the bytes are deterministic. It MUST be standalone, not an entry in `toolsListGoldenSurfaces` — `_DeltaIsEnumerated` reads a frozen `pre099/.json` per listed surface, and direct mode has no pre-feature baseline. +- [ ] T074 In the same gate, assert over `p.directServer.ListTools()` **and** a real `tools/list` driven through the direct server on a session with no agent token, treating any difference between the two sets as a failure. `ListTools()` is the registration map; `handleListTools` serves `filteredTools(ctx)`, and `directServer` is the one routing-mode server carrying `WithToolFilter`s — so registered and served can diverge on exactly this server. Sort by tool name before serializing: `ListTools()` returns a map and Go randomizes iteration order. +- [ ] T075 [P] Add the deferred-direct arm to `bench/arms/` plus its `bench/arms/testdata/*_golden.txt` render golden, following the registry contract in `specs/083-discovery-profiler/contracts/arm-interface.md` and `bench/arms/arm.go`. The existing arms all encode `retrieve_tools` result sets; none renders a direct `tools/list`. +- [ ] T076 Run the SC-001/SC-002 token gates over the frozen 45-tool corpus with the spec-083 profiler's pinned tokenizer: assert ≥70% payload reduction and ≥80% one-shot-callable (non-lossy) share, and record the measured numbers in the PR description. +- [ ] T077 [P] Add E2E coverage in `internal/server/e2e_test.go`: live flip with a connected client (`notifications/tools/list_changed` observed, next listing reflects the mode, tool set identical — SC-006); guessed-wrong → one self-healing retry succeeds (SC-003); and the legacy aliases `/v1/tool_code` and `/v1/tool-code` covered by the **same** mode and notification assertions as `/mcp` (FR-003) — they are easy to miss and are explicitly in the direct-serving set. +- [ ] T078 [P] Run `make swagger` after the config struct change and commit the regenerated `oas/` artifacts. +- [ ] T079 [P] Document `direct_tool_response_mode`, its env alias and its serve flag in `docs/configuration.md`. +- [ ] T080 [P] Write the feature doc under `docs/features/`: the deferral convention (`*`/`~` signature grammar), `describe_tool` on the direct surface, and the client-compat notes — schema-driven form UIs render empty forms; stale cached listings are safe in both directions; the `initialize` instructions delta. +- [ ] T081 [P] Update `CLAUDE.md`: the MCP-protocol built-ins line (`describe_tool` availability on the direct surface) and a Recent Changes entry. +- [ ] T082 Run the full gate set before opening the PR: `go test -race ./internal/...` (using the CI skip regex for `internal/server` locally), `go build -tags server -o /dev/null ./cmd/mcpproxy`, and `/opt/homebrew/bin/golangci-lint run --config .github/.golangci.yml ./...` — the v2 linter CI uses, which is stricter than `scripts/run-linter.sh`. +- [ ] T083 Verify the five frozen gates land as predicted: gates 1 and 3 pass **unregenerated**; gate 2 shows exactly the two enumerated regens; gate 4 passes with the extended `describePlainDelta`; gate 5 is re-measured, not assumed. A diff anywhere else is a regression to fix, not a baseline to update. + +--- + +## Dependencies & execution order + +```text +T004 (fold D1/D2/D3 into spec.md) ──► Phase 1 (config axis + Peek) +T001 (FR-017 atomicity) ──► catalog publication (T017–T020) +T002 (stale schema) ──► Phase 5, and T070/T072's residual cases +T003 (stale annotations) ──► T024, T053 + +Phase 1 ──► Phase 2 (catalog; includes T019a handler capture + T019b describe_tool registration) + │ + ├─► Phase 3 (US1, P1) + ├─► Phase 4 (US2, P1) [T046a completes the T014 collision assertion] + │ + ├─► Phase 5 (US3, P2) [needs T019a's captured entry; independent of US1/US2] + │ │ + │ └─► Phase 6 (US4, P2) ── T070/T072 assert self-healing, which is Phase 5 + │ + └─► Phase 7 (polish) [T073/T074 need Phase 4; T075/T076 generalize T035a] +``` + +Three edges an earlier draft got wrong, all of them cycles or silent gaps: + +- **Phase 2 used to depend on Phase 4.** T024 asserts a fresh proxy lists `describe_tool`, but the registration sat in Phase 4 — so the foundational phase could not go green until the phase built on top of it had landed. Fixed by moving the one-line registration to T019b. +- **Nothing changed `makeDirectModeHandler`'s signature.** T020 updated call sites for a change no task made, and Phase 5 read a captured entry nothing had captured. Fixed by T019a. +- **Phase 6 quietly needed Phase 5.** T070's input-schema residual asserts the stale-schema case self-heals on one retry, and T072 asserts no call succeeds against the wrong schema — both are pre-dispatch-validation behaviour. + +- **US1 and US2 are independent of each other** once Phase 2 lands, and can proceed in parallel by different agents: US1 touches `renderDirectTools` and the instructions helper; US2 touches the describe seam and the direct resolver. They meet only at the shared `buildDirectModeTools` return path (T031 / T037) — sequence those two, or land T037 first since it is a one-line append. +- **US3 depends on Phase 2 only** (the handler's captured catalog entry), not on US1 — it can land before deferral is switchable. +- **US4 depends on there being a rebuild to guard** (Phase 2 T019/T025). + +## Parallel execution examples + +`[P]` means **a different file**, so the marker was removed everywhere it contradicted that — T028/T029/T030 share one new test file, as do T042–T058, T059–T062, and T067/T069. They are still *batchable*, which is not the same thing: + +**Phase 1** — T008 (`loader.go`), T009 (`main.go`), T010 and T011 are genuinely different files and keep `[P]`. T005/T006/T007 all touch `config.go` (or test it) and are sequential. + +**Phase 3 (US1)** — T028, T029, T030 are three cases in one file: write them as one sitting, then T031 makes all three pass. + +**Phase 4 (US2)** — the test tasks all land in `mcp_describe_direct_test.go`: write them as one batch, in the listed order, then the implementation tasks T045–T054 follow sequentially (shared files). + +**Phase 7** — T075, T077, T078, T079, T080, T081 touch six different trees and are genuinely parallel. + +## Implementation strategy + +**MVP = Phase 0 (T004 + T001) + Phase 1 + Phase 2 + Phase 3 + Phase 4**, and it includes **T035a** — the token measurement. Shipping US1 without measuring the reduction would mean shipping the feature without evidence for the one number that justifies it; the earlier draft left that gate in Polish, outside the MVP that claims it. US1 without US2 is not shippable — the spec says so directly ("deferral without the second stage trades tokens for failed calls"), and the catalog-divergence edge case makes an unrecovered deferred listing *strictly worse* than full mode for pending/changed tools. Ship the two P1 stories together. + +**Increment 2**: US3. Turns the worst case from an opaque failure into one bounded retry. Independently valuable — it improves direct mode even with deferral off. + +**Increment 3**: US4. Required before anyone can flip the setting on a live deployment; until then the setting is start-time only. + +**Do not** flip the default in this feature (D8). Any future default flip is its own evidence-gated decision, and the token gates in T076 are the evidence it would need. + +## Notes + +- Total: **88 tasks**. Six were added by cross-model review: T019a (handler signature + entry capture), T019b (describe_tool registration moved out of Phase 4), T035a (Phase 3's own token gate), T046a (collision describe half), T046b (output_schema test before its implementation), and the T063/T064 merge-and-split around activity emission. +- Commits reference `Related #971` — never `Fixes` or `Closes`. +- Local `internal/server` runs need the CI skip regex; a bare `go test ./internal/server` hangs to a 7-minute timeout panic, and `scripts/test-api-e2e.sh` must never be run against a shared machine (it blanket-kills other cores). +- Every task naming a line number in plan.md should re-locate the symbol before editing: the plan's references were accurate at its merge-base and `internal/server/mcp_routing.go` has moved since.