Skip to content

Improve retrieve_tools diagnostics for annotation filters and automation-critical tools #969

Closed

Description

@neylwalecki

Summary

I am using MCPProxy as a local lightweight gateway for many MCP servers, with retrieve_tools as the main way to keep tool schemas out of the agent context until needed. This works very well overall, but I ran into a failure mode where a tool existed and was callable, yet was effectively hidden from routine automation because annotation-based filtering removed it without a very actionable explanation.

I would like to propose a small set of discovery/diagnostic improvements rather than a large architecture change.

Real-world failure mode

A finance automation needed read-only tools from an upstream server, e.g. a tool equivalent to:

wx-copilot:finance_get_capabilities

The upstream server was healthy, tool refresh worked, and direct qualified calls worked. However, retrieve_tools(..., read_only_only=true) did not return the expected finance tools because the upstream tools were missing MCP annotations.readOnlyHint=true.

After the upstream MCP server was fixed to publish annotations like:

{
  "readOnlyHint": true,
  "destructiveHint": false,
  "openWorldHint": true
}

retrieve_tools(..., read_only_only=true) started returning the tools correctly again.

So the root cause was upstream metadata, but from the operator/agent side the experience looked like tool discovery had failed.

Suggested improvements

1. Diagnostics for tools filtered by annotation gates

When a query uses filters such as read_only_only=true, exclude_destructive=true, or exclude_open_world=true, it would be very helpful if the response included a compact diagnostic count, for example:

{
  "filter_diagnostics": {
    "matched_before_filters": 12,
    "omitted_missing_readOnlyHint": 8,
    "omitted_destructiveHint": 2,
    "omitted_openWorldHint": 0,
    "suggestion": "Some tools matched the query but were omitted because readOnlyHint was missing or false. Check upstream annotations or retry without read_only_only for diagnosis."
  }
}

This would make it clear that the tool exists but is being filtered for safety, instead of leaving the user to infer that the server/tool is unavailable.

2. Better upstream-scoped retrieval

A way to scope discovery to a known upstream would reduce ambiguity in large installations:

{
  "query": "finance capabilities",
  "server": "wx-copilot",
  "read_only_only": true
}

or equivalent behavior through query syntax, profile, or server weighting.

This is useful when there are hundreds or thousands of tools and the operator already knows which upstream should satisfy the request.

3. Pinned or required tool IDs for automation preflight

For recurring automations, it would be useful to define a small list of expected tool IDs and have MCPProxy validate their availability and risk metadata explicitly, e.g.:

{
  "required_tools": [
    "wx-copilot:finance_get_capabilities",
    "wx-copilot:finance_search_review_items",
    "wx-copilot:finance_search_obligations",
    "wx-copilot:finance_search_movements"
  ],
  "require_read_only": true
}

This could be a new tool, a mode of retrieve_tools, or a profile-level health check. The goal is not to bypass security. The goal is to fail with a precise explanation when an automation-critical tool is missing, quarantined, unindexed, or lacks required annotations.

Why this fits MCPProxy

I am deliberately not proposing a heavier enterprise gateway model. MCPProxy is valuable to me because it is local, lightweight, uses a single binary/web UI, and focuses on token reduction plus safe discovery. These suggestions seem aligned with that design:

  • Better explanation when safety filters remove tools.
  • Better targeting when the desired upstream is known.
  • Better preflight for automations that depend on a small stable set of tools.

Possible implementation shape

A small first PR could focus only on item 1:

  • Track candidate tools before and after annotation filters.
  • Return compact diagnostic counts when filters remove otherwise relevant matches.
  • Include one short suggestion string only when diagnostic counts are non-zero.

Then upstream-scoped retrieval and required-tool preflight could be discussed separately.

Related context

This is especially important for read-only financial, accounting, operational, or production automations where the operator wants strict filters enabled, but also needs to know whether a missing tool is:

  • Genuinely unavailable.
  • Quarantined.
  • Not indexed yet.
  • Ranked too low.
  • Filtered because upstream annotations are missing or incorrect.

Thanks for building MCPProxy. The retrieval-first model is a strong fit for local multi-MCP workflows; this issue is mostly about making the safe failure modes more legible.

Activity

  1. Dumbris commented on Aug 11, 2026

    @Dumbris
    Member

    Thank you for the excellent report — the concrete field scenario (healthy read-only tools vanishing because the upstream never published readOnlyHint) made this very actionable.

    Item 1 shipped in v0.55.0 (#970, spec 094-filter-diagnostics):

    When read_only_only, exclude_destructive, or exclude_open_world remove tools that matched your query, the retrieve_tools response now includes a compact filter_diagnostics block:

    "filter_diagnostics": {
      "matched_before_filters": 5,
      "omitted_total": 3,
      "omitted_by_filter": {
        "read_only_only": { "missing_annotation": 2, "explicit": 1 }
      },
      "suggestion": "Tools matched but lack upstream annotations required by read_only_only; publish annotations or retry without these filters."
    }

    A few details beyond your sketch:

    • Omissions are split into missing_annotation (fix: publish upstream annotations — your exact case) vs explicit (the filter is working as intended), since the remediations are opposite.
    • The suggestion adapts to the cause and names the responsible filter(s).
    • The block only appears when a filter actually omitted something — responses are byte-identical to before otherwise.
    • Bonus fix your report surfaced: the three filter parameters were missing from the default retrieve_tools schema (only the routing-mode registrations declared them), so agents couldn't discover they existed. All three registrations now share one schema.

    Items 2 (upstream-scoped retrieval) and 3 (required-tools preflight) are deliberately not in this release, per your own "small first PR" suggestion — keeping this issue open to discuss those. Input welcome on which of the two would help your automation more.

  2. neylwalecki commented on Aug 11, 2026

    @neylwalecki
    ContributorAuthor

    Thanks @Dumbris — this is fantastic. The missing_annotation vs explicit split is exactly the kind of distinction that would have made the original incident much easier to diagnose.

    I’ll update and test v0.55.0 locally against the same workflow and report back if I see anything unexpected.

    For our automation use case, item 3 (required_tools / preflight) would help more than item 2 as the next step.

    The reason is that recurring jobs usually know the small set of tool IDs they depend on. What they need most is a deterministic preflight that can fail with a precise reason:

    • tool not found
    • upstream disabled/unhealthy
    • tool quarantined/unapproved
    • not indexed yet
    • missing required annotations, e.g. read-only required but readOnlyHint missing/false
    • explicitly filtered out by risk policy

    Upstream-scoped retrieval is still useful for day-to-day discovery, especially in large installs, but required_tools would directly protect automations from silent discovery misses.

    If useful, I’m happy to help tighten the expected behavior/spec for that flow, or try a small draft PR once the preferred shape is clear.

  3. Dumbris commented on Aug 13, 2026

    @Dumbris
    Member

    Update: item 1 got a full verification pass this week — the filter_diagnostics block behaves per spec across missing_annotation/explicit splits, zero-result cases, and compact mode, so that part is solid ground to build on.

    We're now spec'ing item 3 (the required_tools preflight) with your six failure reasons as the acceptance baseline. Current shape: a shared eligibility evaluator behind a reason-coded verdict per tool (each reason marked retryable / fix-state-first / permanent-config), shipped first as POST /api/v1/preflight + a CLI command with meaningful exit codes — zero MCP tokens spent for CI-style checks.

    One design fork we'd like your input on, since it decides the in-band (MCP) surface: is your automation's preflight moment more like (a) cron/CI — a check that runs before an agent session ever starts, or (b) mid-session — an agent that's already connected over MCP and wants to validate its required toolset before committing to a workflow?

    • If (a), REST + CLI covers you fully and we keep the MCP surface minimal.
    • If (b), we'd add a check-mode to an existing built-in (likely describe_tool, which already has the right visibility pipeline) rather than a new tool, to keep the tools/list token budget flat.

    Two honest scope notes from implementation research: a per-tool "not indexed yet" verdict isn't reliably distinguishable from server startup ordering, so that reason will surface as a server-level server_initializing instead of per-tool; and the preflight will be strictly zero-upstream-I/O (verdicts come from proxy state, never live upstream calls), so a verdict is a statement about what the proxy will do with a call, not a liveness probe of the upstream.

  4. neylwalecki commented on Aug 13, 2026

    @neylwalecki
    ContributorAuthor

    For our case, it is primarily (a) cron/CI. The recurring job knows its small, stable required toolset and should fail deterministically before an agent session spends tokens on discovery or begins the workflow. REST + CLI with meaningful exit codes would cover that baseline well.

    We would still benefit from a thin (b) path later for interactive, mid-session workflows, but only if it reuses the same evaluator and reason codes — ideally as a check mode on an existing built-in rather than a new top-level MCP tool.

    One important boundary for us: keep refresh/liveness separate from preflight. A refresh can be an explicit upstream-I/O operation; the preflight result should describe only the proxy state and eligibility for the next call. server_initializing is a sensible server-level outcome for startup/index ordering, as long as it is clearly distinct from a live upstream health probe.

    The six reason classes and retryability categories sound like the right acceptance baseline. Happy to review a short proposed response schema or test a draft implementation against this scheduled-job flow.

  5. Dumbris commented on Aug 15, 2026

    @Dumbris
    Member

    @neylwalecki — as promised, here is the proposed preflight contract before we ship it. Your cron/CI-first guidance and the "refresh stays separate from preflight" boundary are both baked in. This is being implemented as spec 098-tools-preflight; comments welcome, especially on the reason taxonomy and exit codes your wrappers would branch on.

    Surface (v1)

    REST — POST /api/v1/preflight (X-API-Key or Unix-socket/named-pipe admin; agent tokens supported with scope-silence):

    {
      "tools": [
        {"id": "wx-copilot:finance_get_capabilities"},
        {"id": "gh-ops:sync_issues", "pin_hash": "sha256/v2:9f2c41ab…"}
      ],
      "profile": "nightly-jobs",
      "policy": {"read_only_only": true},
      "wait_ms": 5000
    }

    CLI — mcpproxy tools preflight <id>… [--profile P] [--pin id=hash] [--read-only-only] [--exclude-destructive] [--exclude-open-world] [--wait 5s] [-o json|yaml|table]

    Response

    HTTP 200 whenever the check executed — the verdict is data, never the transport status:

    {
      "verdict": "blocked",
      "checked_at": "2026-08-20T04:00:02Z",
      "tools": [
        {"id": "wx-copilot:finance_get_capabilities", "status": "ready"},
        {
          "id": "gh-ops:sync_issues",
          "status": "unavailable",
          "reason": "tool_changed",
          "retryable": false,
          "action": "approve",
          "detail": "definition changed after approval (hash drift); locked pending review",
          "remediation": "review diff: GET /api/v1/servers/gh-ops/tools/sync_issues/diff"
        }
      ]
    }

    Reason taxonomy (closed, additive-only; unknown code ⇒ treat as non-retryable)

    Class Codes Exit
    retryable (backoff & retry) server_initializing, server_unhealthy 10
    fix-state-first (operator action) server_disabled, server_quarantined, tool_pending_approval, tool_changed, tool_blocked_by_user, oauth_required, hash_mismatch 11
    permanent-config server_not_in_scope†, tool_denied_by_config, missing_annotation, policy_filtered 11
    unknown ID (fix the manifest) not_found (+ scope-filtered did_you_mean), server_not_configured 12

    † operator tier only; agent-token callers see plain not_found (scope-silence preserved).

    CLI exit codes: 0 all ready · 10 retryable-degraded · 11 blocked · 12 unknown ID — worst class wins (12 > 11 > 10), so a wrapper can branch retry-vs-page-vs-fix without parsing JSON. Your six reason classes map: not found → not_found; upstream disabled/unhealthy → server_disabled/server_unhealthy; quarantined/unapproved → server_quarantined/tool_pending_approval/tool_changed; not indexed yet → server_initializing (server-level — during discovery, per-tool existence is genuinely unknowable, so we won't pretend otherwise); missing required annotations → missing_annotation (distinct from policy_filtered = hint explicitly unsafe, since the remediations are opposite — same split filter_diagnostics uses); explicitly filtered by risk policy → policy_filtered/tool_denied_by_config.

    Guarantees per your boundary

    • Stat-only: zero upstream I/O, no reconnects/re-index triggered — verdicts describe proxy state and eligibility for the next call. Refresh/liveness stays a separate explicit operation. Enforced by an instrumented-transport test in CI.
    • wait_ms (optional, cap 10 s): polls local state only for retryable-class failures, always resolves with current reasons at the deadline.
    • pin_hash (optional): rug-pull lockfile semantics — sha256/v{N}:{hex} embeds the hash schema version so a proxy-side hash algorithm upgrade is distinguishable from genuine upstream drift.
    • Transparency: every executed preflight writes an activity record (request-ID correlated, mcpproxy activity list --request-id …) before the 200 is returned.
    • Every reason code ships with a sabotage E2E cell (we deliberately quarantine/disable/hash-drift fixtures and assert the exact code), so the taxonomy is tested, not aspirational.

    The in-band path you flagged for later will be a check mode on the existing describe_tool built-in (same evaluator, same codes, no new top-level tool) in a follow-up spec.

    If this shape works for your scheduled-job flow, a testable build should be available shortly after implementation lands — happy to ping this thread when there's an rc to try.


    Update: this is now implemented and up for review — #996 (feature) with #995 (baseline telemetry counters) landing first. The response shape above is exactly what shipped, plus one refinement found during live verification: not_found can never shadow a blocked/pending/rug-pulled tool (existence is index ∨ approval record), so the reasons you listed survive even after the runtime de-indexes a sabotaged tool. A testable build should be available in the next release — I'll ping here when there's an rc to try against your scheduled-job flow.

  6. 7 remaining items

  7. Dumbris commented on Aug 17, 2026

    @Dumbris
    Member

    @neylwalecki — v0.58.0 is out with the preflight stack, ready to test against your scheduled-job flow: https://github.com/smart-mcp-proxy/mcpproxy-go/releases/tag/v0.58.0

    What's in it for your case:

    • mcpproxy tools preflight <server:tool>… [-o json] [--wait 10s] [--pin id=sha256/v2:…] [--read-only-only] — exit codes 0 ready · 10 retryable · 11 blocked · 12 unknown id (worst class wins), so your cron wrapper branches without parsing JSON.
    • POST /api/v1/preflight — the schema exactly as posted above (verdict is body data, HTTP 200 whenever the check executed; 400 validation / 503 can't-answer-honestly).
    • Bonus beyond the plan: describe_tool {check: true, tool_ids: […]} also landed (the thin in-band path you flagged for later — same evaluator, same codes, batch of 50, verdict-only), so an agent mid-session can gate its own plan too.
    • Your boundary held: preflight is stat-only (zero upstream I/O, enforced by an instrumented-transport test in CI); refresh/liveness stays a separate operation; server_initializing is strictly server-level.
    • Every check writes an activity record before answering — mcpproxy activity list --type preflight or --request-id <id> reconstructs any run.

    One heads-up from the release notes: the describe_tool per-id error code invisible was retired (out-of-scope ids now report not_found) — a disclosure fix; only relevant if you ever branched on that string.

    Your six failure reasons all have dedicated codes and each ships with a sabotage E2E cell (we deliberately quarantine/disable/rug-pull/kill fixtures in CI and assert the exact code). If anything in the real flow surprises you — wrong reason, wrong retryability, anything ambiguous — that's exactly the feedback we want here.

  8. neylwalecki commented on Aug 17, 2026

    @neylwalecki
    ContributorAuthor

    Thanks for the ping — I tested the v0.58.0 macOS ARM64 release in an isolated scheduled-job fixture, and the preflight contract behaved as expected.

    The CLI returned the exact branching signals we need: ready → 0, server_initializing/retryable → 10, missing_annotation and policy_filtered → 11, and not_found → 12. A mixed batch also respected the documented worst-class-wins rule. The REST endpoint returned HTTP 200 with a blocked verdict in the response body, and the corresponding preflight activity records were present.

    The missing_annotation vs policy_filtered split is particularly useful for our automation because it distinguishes an upstream metadata fix from an intentionally unsafe tool. I also ran the focused preflight, CLI, sabotage-matrix, zero-upstream-I/O, and describe_tool check-mode tests from the v0.58.0 tag; they passed.

    After that isolated validation, I updated the operational DMG installation from v0.57.0 to v0.58.0 and ran the preflight against the four stable finance tool IDs from the original scheduled-job case with --read-only-only. All four returned ready and the CLI exited 0. No upstream tool was invoked by the check.

    One small documentation mismatch surfaced: specs/098-tools-preflight/quickstart.md points to internal/testdata/ctl-server.js, while the fixture in the v0.58.0 tag is at internal/server/testdata/preflight_fixture_server.js. Using the latter worked.

    The implementation itself matches the cron/CI boundary we discussed, while the shared describe_tool check path covers the in-session case without adding another top-level tool. Thank you for carrying the original operational case through to a tested release.

  9. Dumbris commented on Aug 17, 2026

    @Dumbris
    Member

    One more pattern worth knowing for your scheduled jobs, since it compounds the token savings: preflight + code execution / stored scripts.

    Instead of the agent orchestrating N tool calls (N schemas + N intermediate results all flowing through model context), gate once and run the whole workflow server-side as one call:

    // 1. one verdict-only check of everything the workflow needs (in-band, or REST/CLI from cron)
    describe_tool {"check": true, "tool_ids": ["wx-copilot:finance_get_capabilities", …]}
    
    // 2. if ready → ONE call runs the whole pipeline in the sandbox
    code_execution {"script": "finance-nightly"}   // scripts/finance-nightly.js next to mcp_config.json

    Inside the script, call_tool() / call_tools() (batched, parallel) do the multi-step work — intermediate payloads never enter model context, only the script's final value returns. With a stored script the code itself stays out of context too: the agent sends 40 bytes, not the program. For an N-step read-only pipeline that's roughly N×(schema+result) worth of context collapsed into one verdict block plus one result value.

    The two features also interlock on the safety side: as of v0.58.0 scripts are policy-gated through the same shared classification as preflight, so a mid-run quarantine flip fails the script with a structured refusal rather than a silent partial run — preflight gives you the pre-run answer, the gate guarantees the in-run one, and both name the same reason.

    Docs: Required-Tools Preflight (composition section) + Code Execution.

  10. Dumbris commented on Aug 17, 2026

    @Dumbris
    Member

    Thanks for the thorough validation — running the tag's own sabotage/zero-I/O suites and the original four-tool case in production is exactly the closure this issue needed.

    The quickstart mismatch you caught is fixed and merged in #1000: the doc now points at the fixture that actually ships (internal/server/testdata/preflight_fixture_server.js) and the recipes were rewritten to be copy-paste runnable against its real knobs (FIXTURE_TOOLS_FILE / FIXTURE_INIT_DELAY_MS / FIXTURE_FAIL_FILE) — updated quickstart.

    With your production validation in, I'm closing this out as delivered (items 1 + 3, plus the in-band path). Item 2 (upstream-scoped retrieval) and the roadmap layers (readyz probe, tool lockfile, declared automation contracts) are tracked for future cycles — new issues welcome any time the scheduled-job flow surfaces something the taxonomy doesn't name cleanly.

  11. Dumbris commented on Sep 2, 2026

    @Dumbris
    Member

    Shipped across three specs.

    • Filter diagnostics (your suggestion 1) landed as spec 094 — retrieve_tools now returns a filter_diagnostics block reporting how many candidates matched before annotation gates and how many each gate removed, with a suggestion string.
    • Pre-dispatch argument validation with self-healing invalid_params errors landed with spec 085.
    • Required-tools preflight and describe_tool check mode landed as specs 098 and 099.

    Thanks for the report — the annotation-metadata failure mode you described is exactly what drove the diagnostics design. Closing; reopen if the shipped output does not cover your case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions