Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .github/workflows/bench-results-untracked.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: Bench Results Untracked

# SC-011 / Spec 065 CN-003: benchmark reports are published as CI artifacts and
# Cloudflare Pages deployments, never committed. bench/results/ is gitignored,
# but .gitignore does not protect a file that is ALREADY tracked (a `git add -f`,
# a rename into the directory, or an ignore rule added after the fact), so the
# rule needs a check that runs on every pull request.
#
# It deliberately does not live in bench.yml: that workflow triggers only on v*
# tags and workflow_dispatch, so it can never block the PR that adds a report.

on:
pull_request:

permissions:
contents: read

jobs:
no-tracked-reports:
name: No benchmark report is tracked
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# `git ls-files`, NOT `git status --porcelain`: porcelain omits ignored
# files, so it would stay silent about exactly the failure this guards —
# a report that is tracked in the index while bench/results/ is ignored.
- name: Assert no report is tracked under bench/results
run: |
tracked="$(git ls-files bench/results)"
if [ -n "$tracked" ]; then
echo "::error::Benchmark reports must never be committed (SC-011, Spec 065 CN-003)."
echo "The following files are tracked under bench/results/:"
echo "$tracked"
echo ""
echo "To fix: git rm --cached <file> and keep bench/results/ gitignored."
echo "Reports are published as CI artifacts and Cloudflare Pages deployments instead."
fi
test -z "$(git ls-files bench/results)"
40 changes: 40 additions & 0 deletions .github/workflows/bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,30 @@ jobs:
- name: Set up uv (uvx-launched MCP reference servers)
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1

# tiktoken (bench/tokens.go) fetches its BPE vocabulary from a remote host
# the first time an encoding is loaded, unless TIKTOKEN_CACHE_DIR points at
# a directory that already holds it. Naming the directory does not fill it,
# so the restore below is the half that actually buys offline reproduction:
# on a hit the vocabulary is on disk before the first bench run, on a miss
# the run fetches it once and the post-job save publishes it for next time.
# Kept under RUNNER_TEMP so the vocabulary never lands in the working tree
# that Cloudflare Pages publishes.
- name: Configure tiktoken vocabulary cache
run: |
mkdir -p "$RUNNER_TEMP/tiktoken"
echo "TIKTOKEN_CACHE_DIR=$RUNNER_TEMP/tiktoken" >> "$GITHUB_ENV"

# Keyed on bench/tokens.go because that file pins the encoding name
# (DefaultEncoding = cl100k_base); the prefix restore-key still serves an
# unrelated edit to that file, at worst re-fetching one vocabulary.
- name: Restore tiktoken vocabulary cache
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ runner.temp }}/tiktoken
key: ${{ runner.os }}-tiktoken-${{ hashFiles('bench/tokens.go') }}
restore-keys: |
${{ runner.os }}-tiktoken-

- name: Build mcpproxy (personal edition)
run: go build -o mcpproxy ./cmd/mcpproxy

Expand Down Expand Up @@ -249,6 +273,22 @@ jobs:
- name: Set up uv (ToolRet parquet fetch)
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1

# Same tiktoken vocabulary cache as the dashboard job above (this job also
# runs the bench binary, so it needs the vocabulary on disk); see there for
# why naming the directory is not enough on its own.
- name: Configure tiktoken vocabulary cache
run: |
mkdir -p "$RUNNER_TEMP/tiktoken"
echo "TIKTOKEN_CACHE_DIR=$RUNNER_TEMP/tiktoken" >> "$GITHUB_ENV"

- name: Restore tiktoken vocabulary cache
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ runner.temp }}/tiktoken
key: ${{ runner.os }}-tiktoken-${{ hashFiles('bench/tokens.go') }}
restore-keys: |
${{ runner.os }}-tiktoken-

# Pinned HF revisions live inside the script; bumping a pin is a
# reviewed event. Cache lands in bench/results/cache/toolret/<revision>/.
- name: Fetch ToolRet at pinned revisions
Expand Down
70 changes: 36 additions & 34 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ graph LR
classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff;
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
class sandbox_isolation,scanner_v2,analytics_dashboard,scanner_simplification,schema_deferred done;
class ux_audit,action_log_transparency,telemetry_identity,telemetry_v7_churn in_progress;
class tpa_db,remote_access_tunnel,token_bench,tool_graph todo;
class ux_audit,action_log_transparency,token_bench,telemetry_identity,telemetry_v7_churn in_progress;
class tpa_db,remote_access_tunnel,tool_graph todo;
```

**Independent epics** (15) — no cross-epic prerequisites; each stands alone:
Expand Down Expand Up @@ -198,6 +198,38 @@ graph LR

</details>

<details>
<summary>🔵 Token-efficiency benchmark: measured savings, published results — In progress · P1</summary>

> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product.

Spec: [103-token-bench](./specs/103-token-bench/) · PR: #1141

```mermaid
graph LR
token_bench_harness["Replay harness: activity-log sessions re-run…"]
token_bench_public["Run public suites locally (τ-bench / BFCL / M…"]
token_bench_blog["Publish results + methodology on mcpproxy.app…"]
token_bench_telemetry["Heartbeat v10: per-tool_response_mode token c…"]

token_bench_harness --> token_bench_public
token_bench_public --> token_bench_blog

classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff;
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
class token_bench_harness in_progress;
class token_bench_public,token_bench_blog,token_bench_telemetry todo;
```

| Task | Status | Refs |
| --- | --- | --- |
| Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries | 🔵 In progress | #1141 |
| Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results | ⚪ Todo | — |
| Publish results + methodology on mcpproxy.app/blog | ⚪ Todo | — |
| Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation | ⚪ Todo | — |

</details>

<details>
<summary>🔵 Telemetry identity &amp; data quality (machine_id + CI-filter hardening) — In progress · P1</summary>

Expand Down Expand Up @@ -349,36 +381,6 @@ graph LR

</details>

<details>
<summary>⚪ Token-efficiency benchmark: measured savings, published results — Todo · P1</summary>

> Measure the real token cost of every routing/savings mode combination — baseline, compact signatures (spec 085), deferred schemas (spec 102), optimistic calling via self-healing pre-dispatch validation, code_execution (spec 096) and stored scripts (spec 097) — on replayed real sessions and on public benchmarks, then publish the results on mcpproxy.app/blog. Every savings number we quote today is an estimate; this turns them into reproducible measurements. Sequenced after schema-deferred so the newest mode is in the matrix. Spec 103 landed 2026-08-31 (#1137 spec+plan, #1139 tasks) after 13 cross-model review rounds; three of its findings changed the design rather than the wording: a recording carries no prompt/conversation/completion oracle so replay CANNOT show agent behaviour (US1 deterministic cost vs US2 live loop are now separate stories); replay needs a FLEET INPUT because the export has no fleet snapshot; and bodies-off yields menu costs plus one cross-mode delta, never an absolute workload cost. The matrix is 5 distinct behaviours, not a 3x2x2 product.

Spec: [103-token-bench](./specs/103-token-bench/)

```mermaid
graph LR
token_bench_harness["Replay harness: activity-log sessions re-run…"]
token_bench_public["Run public suites locally (τ-bench / BFCL / M…"]
token_bench_blog["Publish results + methodology on mcpproxy.app…"]
token_bench_telemetry["Heartbeat v10: per-tool_response_mode token c…"]

token_bench_harness --> token_bench_public
token_bench_public --> token_bench_blog

classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
class token_bench_harness,token_bench_public,token_bench_blog,token_bench_telemetry todo;
```

| Task | Status | Refs |
| --- | --- | --- |
| Replay harness: activity-log sessions re-run under each mode combo; tokens per completed task + first-call success + retries | ⚪ Todo | — |
| Run public suites locally (τ-bench / BFCL / MCP-specific — final list verified by a research pass) and record reproducible results | ⚪ Todo | — |
| Publish results + methodology on mcpproxy.app/blog | ⚪ Todo | — |
| Heartbeat v10: per-tool_response_mode token counters for real-world cohort validation | ⚪ Todo | — |

</details>

<details>
<summary>⚪ Windows native tray app — Todo · P2 · MCP-43</summary>

Expand Down Expand Up @@ -770,12 +772,12 @@ graph LR
| Web UI + macOS app UX audit | In progress | P0 | — | | |
| Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | — | [081-release-qa-gate](./specs/081-release-qa-gate/) | |
| Action log / transparency — info at a glance | In progress | P1 | — | | |
| Token-efficiency benchmark: measured savings, published results | In progress | P1 | 33/64 (52%) | [103-token-bench](./specs/103-token-bench/) | #1141 |
| Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | |
| Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | |
| Planning/docs truth automation | In progress | P2 | — | | |
| Discovery-quality eval harness (Spec 065 second half) | In progress | P3 | — | [065-evaluation-foundation](./specs/065-evaluation-foundation/) | |
| tpa-db: versioned TPA signature database for the offline scanner | Todo | P1 | — | [101-tpa-db](./specs/101-tpa-db/) | |
| Token-efficiency benchmark: measured savings, published results | Todo | P1 | 0/64 (0%) | [103-token-bench](./specs/103-token-bench/) | |
| Windows native tray app `MCP-43` | Todo | P2 | — | | |
| Remote access tunnel (feature-flagged MVP, spec 089) | Todo | P2 | — | [089-remote-access-tunnel](./specs/089-remote-access-tunnel/) | |
| Tool co-occurrence graph (experimental, feature-flagged) | Todo | P2 | — | | |
Expand Down Expand Up @@ -912,4 +914,4 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—`
| [100-prompt-rugpull-baseline](./specs/100-prompt-rugpull-baseline/) | — | — |
| [101-tpa-db](./specs/101-tpa-db/) | — | — |
| [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) |
| [103-token-bench](./specs/103-token-bench/) | `drafted` | 0/64 (0%) |
| [103-token-bench](./specs/103-token-bench/) | `in-flight` | 33/64 (52%) |
Loading
Loading