Skip to content

Commit 38bea56

Browse files
committed
fix(roadmap): tighten two claims from cross-model review round 2
Both findings verified against the tree before applying. 1. The retrieval-D1 blocking scope was stated too narrowly. eval.yml sets continue-on-error only for github.event_name == 'pull_request', and the workflow triggers on pull_request, workflow_dispatch and schedule β€” so the job is report-only on PRs and BLOCKING on both nightly and manual dispatch runs, not "only on the nightly schedule". The note now says so, and records that the workflow's own inline comment carries the same omission. 2. discovery-eval-harness is now done but still declared depends_on: [token-bench-harness], which is unbuilt β€” a done node cannot depend on an open one, and the edge contradicted the corrected note saying both halves shipped independently. Edge removed; the id remains a valid depends_on target for anything that needs it. gen-roadmap.py --check and --check-github both clean.
1 parent dc6890e commit 38bea56

2 files changed

Lines changed: 6 additions & 7 deletions

File tree

β€ŽROADMAP.mdβ€Ž

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,6 @@ graph LR
4848
tool_graph["Tool co-occurrence graph (experimental,…"]
4949
telemetry_identity["Telemetry identity & data quality (mach…"]
5050
telemetry_v7_churn["Telemetry v7: honest funnel + churn ins…"]
51-
discovery_eval_harness["Discovery-quality eval harness (Spec 06…"]
5251
5352
scanner_v2 --> sandbox_isolation
5453
ux_audit --> action_log_transparency
@@ -61,17 +60,16 @@ graph LR
6160
schema_deferred --> token_bench
6261
analytics_dashboard --> tool_graph
6362
telemetry_identity --> telemetry_v7_churn
64-
token_bench --> discovery_eval_harness
6563
6664
classDef done fill:#1f7a1f,stroke:#0d3d0d,color:#ffffff;
6765
classDef in_progress fill:#1f6feb,stroke:#0b3d91,color:#ffffff;
6866
classDef todo fill:#6e7781,stroke:#3d4248,color:#ffffff;
69-
class sandbox_isolation,scanner_v2,analytics_dashboard,scanner_simplification,schema_deferred,discovery_eval_harness done;
67+
class sandbox_isolation,scanner_v2,analytics_dashboard,scanner_simplification,schema_deferred done;
7068
class ux_audit,action_log_transparency,telemetry_identity,telemetry_v7_churn in_progress;
7169
class tpa_db,remote_access_tunnel,token_bench,tool_graph todo;
7270
```
7371

74-
**Independent epics** (14) β€” no cross-epic prerequisites; each stands alone:
72+
**Independent epics** (15) β€” no cross-epic prerequisites; each stands alone:
7573

7674
- πŸ”΅ **Release qualification gate (auto-QA matrix blocks the tag)** β€” In progress Β· P0
7775
- πŸ”΅ **Planning/docs truth automation** β€” In progress Β· P2
@@ -87,6 +85,7 @@ graph LR
8785
- 🟒 **Connect step trust: preview, visible backup, one-click undo** β€” Done Β· P0
8886
- 🟒 **Registries β€” easier search + add-server** β€” Done Β· P1
8987
- 🟒 **Tray↔core decoupling: socket/REST API only, no config-file reads** β€” Done Β· P2
88+
- 🟒 **Discovery-quality eval harness (Spec 065 second half)** β€” Done Β· P3
9089

9190
## Epic details
9291

@@ -743,7 +742,7 @@ graph LR
743742
<details>
744743
<summary>🟒 Discovery-quality eval harness (Spec 065 second half) β€” Done Β· P3</summary>
745744

746-
> DONE β€” 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo β€” note it is REPORT-ONLY on pull requests (continue-on-error, npx/uvx fetch flake) and blocks only on the nightly schedule, so promoting it to PR-blocking after a green soak is still open (MCP-742). A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness.
745+
> DONE β€” 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo β€” note continue-on-error is scoped to github.event_name == 'pull_request', so the job is REPORT-ONLY on PRs (npx/uvx fetch flake) and BLOCKING on both the nightly schedule and manual workflow_dispatch runs. Promoting it to PR-blocking after a green soak is still open (MCP-742). NB the workflow's own inline comment says 'blocking on the nightly schedule' and omits workflow_dispatch. A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness.
747746

748747
Spec: [065-evaluation-foundation](./specs/065-evaluation-foundation/)
749748

β€Žroadmap.yamlβ€Ž

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -728,5 +728,5 @@ epics:
728728
status: done
729729
priority: P3
730730
spec: specs/065-evaluation-foundation
731-
depends_on: [token-bench-harness]
732-
note: "DONE β€” 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo β€” note it is REPORT-ONLY on pull requests (continue-on-error, npx/uvx fetch flake) and blocks only on the nightly schedule, so promoting it to PR-blocking after a green soak is still open (MCP-742). A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness."
731+
depends_on: []
732+
note: "DONE β€” 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo β€” note continue-on-error is scoped to github.event_name == 'pull_request', so the job is REPORT-ONLY on PRs (npx/uvx fetch flake) and BLOCKING on both the nightly schedule and manual workflow_dispatch runs. Promoting it to PR-blocking after a green soak is still open (MCP-742). NB the workflow's own inline comment says 'blocking on the nightly schedule' and omits workflow_dispatch. A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness."

0 commit comments

Comments
Β (0)