Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,6 @@ tail -f ~/Library/Logs/mcpproxy/main.log # main log (macOS; Linux: ~/.mcpproxy/
- **Windows installer**: [docs/github-actions-windows-wix-research.md](docs/github-actions-windows-wix-research.md). **Prerelease** (`next` branch + `v*-rc.*` tags, opt-in, off stable channels): [docs/prerelease-builds.md](docs/prerelease-builds.md).

## Recent Changes
- 103-token-bench: Go 1.25 (`bench/`) + existing only — tiktoken-go v0.1.8 (cl100k_base), the mcp-go transport already used by `bench/mcpcaller.go`. MCPMark is an external SHA-pinned tool invoked out of process, not a module dependency. **No new dependencies.**
- 102-schema-deferred: Go 1.25.5 module toolchain (`go.mod`), backend-only + existing only — mark3labs/mcp-go v0.57.0 (tool surface), santhosh-tekuri/jsonschema/v6 (Spec-085 validator reuse), zap, stdlib. **No new dependencies.**
- 101-tpa-db: Go 1.25.5 module toolchain (`go.mod`), backend-only + stdlib only for the new work — `crypto/ed25519`, `crypto/sha256`, `syscall` (no ed25519 usage in the tree before this); existing bbolt (state), zap, Cobra. **No new dependencies.**
- 098-tools-preflight: Added Go 1.24 module toolchain (repo builds with local Go 1.25) + existing only — chi (httpapi), bbolt (storage), Bleve (index), zap (logging), Cobra (CLI), swaggo/swag v2 (contract regen). **No new dependencies.**
- 097-stored-scripts: Added Go 1.25 (os.Root/Root.ReadFile available — R1) + stdlib only (os.Root). **No new dependencies.**
119 changes: 119 additions & 0 deletions specs/103-token-bench/checklists/requirements.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Specification Quality Checklist: Token-efficiency benchmark — measured savings, published results

**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2026-08-31
**Feature**: [spec.md](../spec.md)

## Content Quality

- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed

## Requirement Completeness

- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified

## Feature Readiness

- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification

## Notes

Validation ran over three iterations; the following were fixed rather than waived.

**Iteration 1 — implementation detail leaked into a spec that must stay declarative.**
The draft named concrete files, flags, CLI commands, struct fields and PR numbers
(`bench/session.go`, `armRetryRates`, `ResponseTruncated`, `-corpus-v2`, `#747`). All were
removed from requirements and success criteria. This was the hardest constraint to hold,
because the feature's defining risk — duplicating a harness that already exists — is
naturally expressed in file names. It is instead expressed as the **Scope Boundary** table
in capability terms, which a non-implementer can still act on.

**Iteration 2 — success criteria were not all verifiable without implementation knowledge.**
Rewritten so each states an outcome a reader can check: SC-003 (an outsider reproduces the
numbers), SC-006 (the benchmark detects a false saving), SC-007 (truncated input is never
counted silently).

**Iteration 3 — scope was bounded but the *anti*-scope was not.**
Added Out of Scope, because the likeliest failure of this feature is not under-delivery but
sprawl into optimisation work the measurement merely suggests.

**No [NEEDS CLARIFICATION] markers were used.** Three candidates were resolved by informed
default and recorded in Assumptions instead:
- *Which model to pin for agent-loop runs* — deferred to plan stage as a budget decision,
and the deterministic half is specified to stand alone without it (FR-015 constrains how
results are reported, not which model produces them).
- *Whether a completion signal can be derived from recorded sessions* — assumed derivable,
with an explicit fallback (report sessions lacking the signal rather than assuming
success). This is the assumption most likely to be wrong and should be validated first at
plan stage.
- *Which public suite to adopt* — deliberately left as a capability requirement (FR-020:
"at least one full agent-loop suite") rather than naming one, because the shortlist was
researched in 2026-08 and needs re-confirmation against the suites' current code before
being committed to.

**Carried into planning**: the research pass flagged two items as unverified — whether the
candidate suites emit per-task token counts, and whether they can be pointed at a single
proxy endpoint. FR-020 and FR-022 are unsatisfiable if neither holds, so plan stage must
read the suites' code before committing to one.

## Cross-model review round 1 (opencode gpt-5.6-sol, 2026-08-31)

Seven findings, all verified against the tree and all applied. Two were **Critical and
changed the spec's premise**:

1. **Replay cannot answer what the spec asked it to.** The activity record carries tool
calls, arguments, responses, status and work-session grouping — but no prompt, no
conversation, no model state and no completion oracle (`internal/storage/activity_models.go`).
So replaying a recording under a different mode cannot reveal how the agent would have
*behaved*: which calls it would attempt, whether the first was right, whether it finished.
The original US1 asked for exactly that and was unsatisfiable.
**Fix**: added a binding **Replay Boundary** section and split the old US1 into US1
(deterministic counterfactual cost recomputation over real workload shape, no model) and
US2 (live agent loop under a pinned model, which owns every success/completion figure).
FR-004 now forbids inferring behaviour from a recording, and Out of Scope names it.

2. **Reproducibility and privacy were in direct conflict, and the privacy side was absent.**
The body-inclusive export deliberately returns full unmasked values including detected
secrets — it is the compliance surface, not a browsing one (`internal/httpapi/activity.go`).
The spec said session contents must stay out of published artefacts but said nothing about
protecting them *during* replay, including transmission to model providers.
**Fix**: new FR-006..FR-009 (sensitive by default, never transmitted to third parties,
flagged records excluded or reduced, inputs live outside the repo with a documented
deletion step) plus SC-009. FR-030 now admits that figures from private sessions are not
independently reproducible and cannot be the sole support for a published claim.

Also applied:

3. **The headline metric did not actually penalise worse completion.** Tokens-per-completed-task
can *improve* while completion falls, if tokens fall faster — so the old SC-006 was not
generally true. **Fix**: FR-018 requires completion rate at equal prominence, FR-019 adds a
completion-regression threshold that overrides any token saving, and SC-007 now requires
demonstrating this with a deliberately degraded mode.
4. **SC-001 demanded results for "every mode combination"** while FR-017 requires skipping
invalid ones. **Fix**: scoped to every *valid* combination exercised by the live task set.
5. **Core terms were undefined** ("first attempt", "accepted", "retry", "unit of work",
"completion", token accounting source). **Fix**: a binding Definitions block, which also
separates corrective retries from infrastructure retries.
6. **Three FRs restated existing capabilities.** **Fix**: moved to an Inherited Constraints
block (IC-001..IC-004) so they are satisfied, not rebuilt.
7. **The capability axes were not a testable matrix.** **Fix**: FR-016 states them as binary
conditions over the routing modes where each is available, with the report enumerating
applicable rows.

The reviewer independently confirmed the spec's current-state claims: nothing in the harness
replays activity exports, retry rates are literature-derived defaults, first-call success is
not measured, the three axes are not crossed, and the spec-102 figures (29.7% / 34.8% /
38.9% ceiling vs a 70% projection) are accurate.
89 changes: 89 additions & 0 deletions specs/103-token-bench/contracts/mode-matrix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Contract: the mode matrix

**5 distinct behaviours + 1 baseline. The 3-axis product's other 7 combinations are
configurable but redundant, each a skip-with-reason
row.** This file is the source of truth for FR-015, FR-016 and FR-017.

## Valid cells

| id | Endpoint | `tool_response_mode` | `direct_tool_response_mode` | Requires |
|----|----------|----------------------|------------------------------|----------|
| `retrieve_full` | retrieve_tools surface | `full` | n/a | — (default) |
| `retrieve_compact` | retrieve_tools surface | `compact` | n/a | spec 085 |
| `direct_full` | direct surface | n/a | `full` | — |
| `direct_deferred` | direct surface | n/a | `deferred` | spec 102 |
| `code_exec` | code-execution surface | forced `full` | n/a | `enable_code_execution: true` |
| `baseline` | none — all upstream tools loaded directly | n/a | n/a | FR-020 denominator |

`baseline` is not an mcpproxy mode. It is the same agent doing the same tasks with every
upstream tool inline, and it is what every percentage is measured against.

## The other 7 combinations of the 3-axis product (FR-017)

The naive product is 3 routing modes x 2 discovery serializations x 2 direct serializations =
**12 combinations, which collapse onto 5 distinct behaviours.**

They are **not impossible to configure** — they are *configurable and behaviourally redundant*,
because on each surface one or both serialization axes have no consumer. Calling them
"impossible" would be wrong; rendering them as zeros would be worse. Each is a
**skip-with-reason row naming the cell it collapses onto.**

| Configured combination | Collapses onto | Reason code |
|---|---|---|
| `retrieve_tools` x full x **deferred** | `retrieve_full` | `axis-ignored` |
| `retrieve_tools` x compact x **deferred** | `retrieve_compact` | `axis-ignored` |
| `direct` x **compact** x full | `direct_full` | `axis-ignored` |
| `direct` x **compact** x deferred | `direct_deferred` | `axis-ignored` |
| `code_execution` x **compact** x full | `code_exec` | `forced-full` |
| `code_execution` x full x **deferred** | `code_exec` | `axis-ignored` |
| `code_execution` x **compact** x **deferred** | `code_exec` | `forced-full` + `axis-ignored` |

### Why each axis is ignored where it is

- **`tool_response_mode` governs exactly one surface**: resolved by
`effectiveToolResponseMode`, whose single production call is in the `retrieve_tools` handler
(`internal/server/mcp.go:1584-1585`). Neither the direct nor the code-execution surface reads
it — the direct surface has no `retrieve_tools` tool at all.
- **`direct_tool_response_mode` governs exactly one surface**: resolved by
`effectiveDirectToolResponseMode`. Note it has THREE production call sites, not one —
building the listing (`mcp_routing.go:100`), detecting serialization drift (`:1040-1045`)
and logging a reload (`:1071-1076`) — but all three concern the direct surface only, so the
collapse holds. The claim is one *surface*, not one *call site*.
- **`code_execution` forces full**: that surface overwrites the response mode with `full` and
blanks the detail parameter; `detail` is not in its schema. That is a genuine *override*,
distinct from an ignored axis, hence its own reason code.

### Separately: a degenerate configuration

`code_execution` with `enable_code_execution: false` is not part of the product above. It is a
configuration in which the surface can discover tools and call none of them. Skip it with
reason `degenerate`.

## Rules

1. **The routing-mode axis is selected by endpoint URL — no config change, no restart.** All
three routing-mode servers are built at startup and all three endpoints stay mounted
regardless of config (`internal/server/server.go:2514-2536`).
2. **The two serialization axes DO require config**, and each affects only its own surface.
A cell is therefore (URL, serialization-config) — **not URL alone**.
**Both hot-reload, so the matrix still crosses on ONE long-lived instance**, with a config
apply between serialization cells: `tool_response_mode` is read from the live snapshot on
every call (`internal/server/profile_resolver.go:49-64`, documented as taking effect "on
the very next call, without reconstructing the server"), and `direct_tool_response_mode` is
rebuilt on the `config.reloaded` event (`internal/server/server.go:569-597`).
3. **An ignored axis is recorded as not-applicable**, never as a default value — a
default would imply a measurement that was never taken.
4. **A skipped row carries a reason code AND the cell it collapses onto, and never renders
as a zero.** Reuse the existing
skipped-row shape (`Skipped` / `SkipReason` plus the skipped-result constructor) rather
than inventing a second one.
5. **Capability toggles** (batching, stored scripts, validate-before-dispatch) are binary
conditions applied to the cells where each is available, and the report enumerates which
cells each applies to. They are not additional axes of a product.
6. **Cell ids are stable** across runs and releases, so results remain comparable (FR-028).

## Fleet shaping

Every cell must be measured against the SAME fleet in a given comparison, and the fleet must
be large enough for proxy modes to differ from baseline. A single upstream server's toolset is
too small to show the asymptote; load the full fleet even when exercising one service's tasks.
Loading
Loading