You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 38bea56
Browse filesBrowse the repository at this point in the historyBrowse files
fix(roadmap): tighten two claims from cross-model review round 2
Both findings verified against the tree before applying.
1. The retrieval-D1 blocking scope was stated too narrowly. eval.yml sets
continue-on-error only for github.event_name == 'pull_request', and the
workflow triggers on pull_request, workflow_dispatch and schedule β so the
job is report-only on PRs and BLOCKING on both nightly and manual dispatch
runs, not "only on the nightly schedule". The note now says so, and records
that the workflow's own inline comment carries the same omission.
2. discovery-eval-harness is now done but still declared
depends_on: [token-bench-harness], which is unbuilt β a done node cannot
depend on an open one, and the edge contradicted the corrected note saying
both halves shipped independently. Edge removed; the id remains a valid
depends_on target for anything that needs it.
gen-roadmap.py --check and --check-github both clean.
> DONE β 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo β note it is REPORT-ONLY on pull requests (continue-on-error, npx/uvx fetch flake) and blocks only on the nightly schedule, so promoting it to PR-blocking after a green soak is still open (MCP-742). A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness.
745
+
> DONE β 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo β note continue-on-error is scoped to github.event_name == 'pull_request', so the job is REPORT-ONLY on PRs (npx/uvx fetch flake) and BLOCKING on both the nightly schedule and manual workflow_dispatch runs. Promoting it to PR-blocking after a green soak is still open (MCP-742). NB the workflow's own inline comment says 'blocking on the nightly schedule' and omits workflow_dispatch. A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness.
Copy file name to clipboardExpand all lines: roadmap.yaml
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -728,5 +728,5 @@ epics:
728
728
status: done
729
729
priority: P3
730
730
spec: specs/065-evaluation-foundation
731
-
depends_on: [token-bench-harness]
732
-
note: "DONE β 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo β note it is REPORT-ONLY on pull requests (continue-on-error, npx/uvx fetch flake) and blocks only on the nightly schedule, so promoting it to PR-blocking after a green soak is still open (MCP-742). A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness."
731
+
depends_on: []
732
+
note: "DONE β 2026-08-31 audit: both halves shipped INDEPENDENTLY, not via token-bench-harness. The earlier 'superseded / folded into token-bench-harness' framing was wrong on its own terms: token-bench-harness is still unbuilt, so nothing could have been folded into it. Security recall/FP half: cmd/scan-eval, backing the Spec 076/077 gate in eval.yml. Discovery-quality half: the eval.yml retrieval-d1 job boots mcpproxy and scores retrieval_golden_v1.json against a committed baseline at --tolerance 0.05 via the pinned external mcp-eval repo β note continue-on-error is scoped to github.event_name == 'pull_request', so the job is REPORT-ONLY on PRs (npx/uvx fetch flake) and BLOCKING on both the nightly schedule and manual workflow_dispatch runs. Promoting it to PR-blocking after a green soak is still open (MCP-742). NB the workflow's own inline comment says 'blocking on the nightly schedule' and omits workflow_dispatch. A second in-repo implementation lives in bench/: metrics.go defines RecallAtK/NDCGAtK, and the SC-003 recall@5 = 0.68 +/- 0.05 parity gate through the production Bleve index is asserted in bench/armindex_test.go (armindex.go supplies the production index wiring, not the assertion). Kept as a stable depends_on target; do not build a standalone harness."
0 commit comments