Skip to content

smoke-invariants: EXPECTED_TOOLS omits check_index_coverage and compare_graphs, and the check is one-way #1969

Closed

Description

@CaptainMittens

scripts/smoke-invariants.sh asserts that the server advertises every expected tool. Its list omits two tools the server does register:

EXPECTED_TOOLS="index_repository search_graph query_graph trace_path get_code_snippet \
get_graph_schema get_architecture search_code list_projects delete_project index_status \
detect_changes manage_adr ingest_traces"
EXPECTED_TOOL_COUNT=14

check_index_coverage and compare_graphs are missing. Both are registered in src/mcp/mcp.c (the tool table, and the analysis profile allowlist). Found while working on #1941; not fixed there.

Why the check does not catch it

inv_tools_list walks the expected list and reports the names it did not find in the server's tools/list response. It never walks the other direction. A tool the server advertises but the list does not name is simply invisible, so the invariant passes while covering 14 of 16 tools and prints all 14 tools present — a number that reads like full coverage.

That is the same failure the parse-coverage work in #1941 was about: a check that reports success without having looked at everything.

Suggested fix

Add the two names and set the count to 16, then make the check symmetric — report any advertised tool the list does not name, so the next tool added cannot go unnoticed the same way. EXPECTED_TOOL_COUNT should be derived from EXPECTED_TOOLS rather than written twice.

Note that smoke-invariants.sh runs from smoke.yml, whose triggers are workflow_dispatch and pushes to qa/smoke-**, and which is documented non-gating. So this is a gap in a check that never runs on a PR.

Related: #963, #1941.

Activity

  1. added
    bugSomething isn't working
    priority/highNeeds near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.
    github_actionsPull requests that update GitHub Actions code
    on Sep 5, 2026
  2. DeusData commented on Sep 19, 2026

    @DeusData
    Owner

    Thank you for identifying the registry drift. Current main still hard-codes 14 expected tools in scripts/smoke-invariants.sh and omits check_index_coverage and compare_graphs, both present in the MCP tool registry. One detail has changed: the smoke test now checks the returned count as well as expected-name membership, so describing the current check as purely one-way would be inaccurate. The remaining work is to reconcile the expected inventory with the intended tool profile and prove missing, extra, and duplicate-name failures. This is source confirmation; I have not run the smoke test in this triage pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggithub_actionsPull requests that update GitHub Actions codeparsing/qualityGraph extraction bugs, false positives, missing edgespriority/highNeeds near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions