Repository navigation
Description
Summary
I am using MCPProxy as a local lightweight gateway for many MCP servers, with retrieve_tools as the main way to keep tool schemas out of the agent context until needed. This works very well overall, but I ran into a failure mode where a tool existed and was callable, yet was effectively hidden from routine automation because annotation-based filtering removed it without a very actionable explanation.
I would like to propose a small set of discovery/diagnostic improvements rather than a large architecture change.
Real-world failure mode
A finance automation needed read-only tools from an upstream server, e.g. a tool equivalent to:
wx-copilot:finance_get_capabilities
The upstream server was healthy, tool refresh worked, and direct qualified calls worked. However, retrieve_tools(..., read_only_only=true) did not return the expected finance tools because the upstream tools were missing MCP annotations.readOnlyHint=true.
After the upstream MCP server was fixed to publish annotations like:
{
"readOnlyHint": true,
"destructiveHint": false,
"openWorldHint": true
}retrieve_tools(..., read_only_only=true) started returning the tools correctly again.
So the root cause was upstream metadata, but from the operator/agent side the experience looked like tool discovery had failed.
Suggested improvements
1. Diagnostics for tools filtered by annotation gates
When a query uses filters such as read_only_only=true, exclude_destructive=true, or exclude_open_world=true, it would be very helpful if the response included a compact diagnostic count, for example:
{
"filter_diagnostics": {
"matched_before_filters": 12,
"omitted_missing_readOnlyHint": 8,
"omitted_destructiveHint": 2,
"omitted_openWorldHint": 0,
"suggestion": "Some tools matched the query but were omitted because readOnlyHint was missing or false. Check upstream annotations or retry without read_only_only for diagnosis."
}
}This would make it clear that the tool exists but is being filtered for safety, instead of leaving the user to infer that the server/tool is unavailable.
2. Better upstream-scoped retrieval
A way to scope discovery to a known upstream would reduce ambiguity in large installations:
{
"query": "finance capabilities",
"server": "wx-copilot",
"read_only_only": true
}or equivalent behavior through query syntax, profile, or server weighting.
This is useful when there are hundreds or thousands of tools and the operator already knows which upstream should satisfy the request.
3. Pinned or required tool IDs for automation preflight
For recurring automations, it would be useful to define a small list of expected tool IDs and have MCPProxy validate their availability and risk metadata explicitly, e.g.:
{
"required_tools": [
"wx-copilot:finance_get_capabilities",
"wx-copilot:finance_search_review_items",
"wx-copilot:finance_search_obligations",
"wx-copilot:finance_search_movements"
],
"require_read_only": true
}This could be a new tool, a mode of retrieve_tools, or a profile-level health check. The goal is not to bypass security. The goal is to fail with a precise explanation when an automation-critical tool is missing, quarantined, unindexed, or lacks required annotations.
Why this fits MCPProxy
I am deliberately not proposing a heavier enterprise gateway model. MCPProxy is valuable to me because it is local, lightweight, uses a single binary/web UI, and focuses on token reduction plus safe discovery. These suggestions seem aligned with that design:
- Better explanation when safety filters remove tools.
- Better targeting when the desired upstream is known.
- Better preflight for automations that depend on a small stable set of tools.
Possible implementation shape
A small first PR could focus only on item 1:
- Track candidate tools before and after annotation filters.
- Return compact diagnostic counts when filters remove otherwise relevant matches.
- Include one short suggestion string only when diagnostic counts are non-zero.
Then upstream-scoped retrieval and required-tool preflight could be discussed separately.
Related context
This is especially important for read-only financial, accounting, operational, or production automations where the operator wants strict filters enabled, but also needs to know whether a missing tool is:
- Genuinely unavailable.
- Quarantined.
- Not indexed yet.
- Ranked too low.
- Filtered because upstream annotations are missing or incorrect.
Thanks for building MCPProxy. The retrieval-first model is a strong fit for local multi-MCP workflows; this issue is mostly about making the safe failure modes more legible.
Activity
test(server): close two vacuous-assertion gaps in the diagnostics tes…
test(server): pin spec094 goldens to LF so Windows byte-identity holds
feat(server): retrieve_tools filter diagnostics (spec 094) (#970)
Dumbris commentedon Aug 11, 2026
Thank you for the excellent report — the concrete field scenario (healthy read-only tools vanishing because the upstream never published
readOnlyHint) made this very actionable.Item 1 shipped in v0.55.0 (#970, spec
094-filter-diagnostics):When
read_only_only,exclude_destructive, orexclude_open_worldremove tools that matched your query, theretrieve_toolsresponse now includes a compactfilter_diagnosticsblock:A few details beyond your sketch:
missing_annotation(fix: publish upstream annotations — your exact case) vsexplicit(the filter is working as intended), since the remediations are opposite.retrieve_toolsschema (only the routing-mode registrations declared them), so agents couldn't discover they existed. All three registrations now share one schema.Items 2 (upstream-scoped retrieval) and 3 (required-tools preflight) are deliberately not in this release, per your own "small first PR" suggestion — keeping this issue open to discuss those. Input welcome on which of the two would help your automation more.
neylwalecki commentedon Aug 11, 2026
Thanks @Dumbris — this is fantastic. The
missing_annotationvsexplicitsplit is exactly the kind of distinction that would have made the original incident much easier to diagnose.I’ll update and test v0.55.0 locally against the same workflow and report back if I see anything unexpected.
For our automation use case, item 3 (
required_tools/ preflight) would help more than item 2 as the next step.The reason is that recurring jobs usually know the small set of tool IDs they depend on. What they need most is a deterministic preflight that can fail with a precise reason:
readOnlyHintmissing/falseUpstream-scoped retrieval is still useful for day-to-day discovery, especially in large installs, but
required_toolswould directly protect automations from silent discovery misses.If useful, I’m happy to help tighten the expected behavior/spec for that flow, or try a small draft PR once the preferred shape is clear.
Dumbris commentedon Aug 13, 2026
Update: item 1 got a full verification pass this week — the
filter_diagnosticsblock behaves per spec across missing_annotation/explicit splits, zero-result cases, and compact mode, so that part is solid ground to build on.We're now spec'ing item 3 (the
required_toolspreflight) with your six failure reasons as the acceptance baseline. Current shape: a shared eligibility evaluator behind a reason-coded verdict per tool (each reason marked retryable / fix-state-first / permanent-config), shipped first asPOST /api/v1/preflight+ a CLI command with meaningful exit codes — zero MCP tokens spent for CI-style checks.One design fork we'd like your input on, since it decides the in-band (MCP) surface: is your automation's preflight moment more like (a) cron/CI — a check that runs before an agent session ever starts, or (b) mid-session — an agent that's already connected over MCP and wants to validate its required toolset before committing to a workflow?
describe_tool, which already has the right visibility pipeline) rather than a new tool, to keep the tools/list token budget flat.Two honest scope notes from implementation research: a per-tool "not indexed yet" verdict isn't reliably distinguishable from server startup ordering, so that reason will surface as a server-level
server_initializinginstead of per-tool; and the preflight will be strictly zero-upstream-I/O (verdicts come from proxy state, never live upstream calls), so a verdict is a statement about what the proxy will do with a call, not a liveness probe of the upstream.neylwalecki commentedon Aug 13, 2026
For our case, it is primarily (a) cron/CI. The recurring job knows its small, stable required toolset and should fail deterministically before an agent session spends tokens on discovery or begins the workflow. REST + CLI with meaningful exit codes would cover that baseline well.
We would still benefit from a thin (b) path later for interactive, mid-session workflows, but only if it reuses the same evaluator and reason codes — ideally as a check mode on an existing built-in rather than a new top-level MCP tool.
One important boundary for us: keep refresh/liveness separate from preflight. A refresh can be an explicit upstream-I/O operation; the preflight result should describe only the proxy state and eligibility for the next call. server_initializing is a sensible server-level outcome for startup/index ordering, as long as it is clearly distinct from a live upstream health probe.
The six reason classes and retryability categories sound like the right acceptance baseline. Happy to review a short proposed response schema or test a draft implementation against this scheduled-job flow.
Dumbris commentedon Aug 15, 2026
@neylwalecki — as promised, here is the proposed preflight contract before we ship it. Your cron/CI-first guidance and the "refresh stays separate from preflight" boundary are both baked in. This is being implemented as spec
098-tools-preflight; comments welcome, especially on the reason taxonomy and exit codes your wrappers would branch on.Surface (v1)
REST —
POST /api/v1/preflight(X-API-Key or Unix-socket/named-pipe admin; agent tokens supported with scope-silence):{ "tools": [ {"id": "wx-copilot:finance_get_capabilities"}, {"id": "gh-ops:sync_issues", "pin_hash": "sha256/v2:9f2c41ab…"} ], "profile": "nightly-jobs", "policy": {"read_only_only": true}, "wait_ms": 5000 }CLI —
mcpproxy tools preflight <id>… [--profile P] [--pin id=hash] [--read-only-only] [--exclude-destructive] [--exclude-open-world] [--wait 5s] [-o json|yaml|table]Response
HTTP 200 whenever the check executed — the verdict is data, never the transport status:
{ "verdict": "blocked", "checked_at": "2026-08-20T04:00:02Z", "tools": [ {"id": "wx-copilot:finance_get_capabilities", "status": "ready"}, { "id": "gh-ops:sync_issues", "status": "unavailable", "reason": "tool_changed", "retryable": false, "action": "approve", "detail": "definition changed after approval (hash drift); locked pending review", "remediation": "review diff: GET /api/v1/servers/gh-ops/tools/sync_issues/diff" } ] }Reason taxonomy (closed, additive-only; unknown code ⇒ treat as non-retryable)
server_initializing,server_unhealthyserver_disabled,server_quarantined,tool_pending_approval,tool_changed,tool_blocked_by_user,oauth_required,hash_mismatchserver_not_in_scope†,tool_denied_by_config,missing_annotation,policy_filterednot_found(+ scope-filtereddid_you_mean),server_not_configured† operator tier only; agent-token callers see plain
not_found(scope-silence preserved).CLI exit codes:
0all ready ·10retryable-degraded ·11blocked ·12unknown ID — worst class wins (12 > 11 > 10), so a wrapper can branch retry-vs-page-vs-fix without parsing JSON. Your six reason classes map: not found →not_found; upstream disabled/unhealthy →server_disabled/server_unhealthy; quarantined/unapproved →server_quarantined/tool_pending_approval/tool_changed; not indexed yet →server_initializing(server-level — during discovery, per-tool existence is genuinely unknowable, so we won't pretend otherwise); missing required annotations →missing_annotation(distinct frompolicy_filtered= hint explicitly unsafe, since the remediations are opposite — same split filter_diagnostics uses); explicitly filtered by risk policy →policy_filtered/tool_denied_by_config.Guarantees per your boundary
wait_ms(optional, cap 10 s): polls local state only for retryable-class failures, always resolves with current reasons at the deadline.pin_hash(optional): rug-pull lockfile semantics —sha256/v{N}:{hex}embeds the hash schema version so a proxy-side hash algorithm upgrade is distinguishable from genuine upstream drift.mcpproxy activity list --request-id …) before the 200 is returned.The in-band path you flagged for later will be a
checkmode on the existingdescribe_toolbuilt-in (same evaluator, same codes, no new top-level tool) in a follow-up spec.If this shape works for your scheduled-job flow, a testable build should be available shortly after implementation lands — happy to ping this thread when there's an rc to try.
Update: this is now implemented and up for review — #996 (feature) with #995 (baseline telemetry counters) landing first. The response shape above is exactly what shipped, plus one refinement found during live verification:
not_foundcan never shadow a blocked/pending/rug-pulled tool (existence is index ∨ approval record), so the reasons you listed survive even after the runtime de-indexes a sabotaged tool. A testable build should be available in the next release — I'll ping here when there's an rc to try against your scheduled-job flow.fix(telemetry): correct preflight counter attribution from cross-mode…
fix(telemetry): order diagnostics notes by delivery, require an intac…
test(telemetry): pin the diagnostics brace scanner's escape handling
fix(telemetry): derive availability_block_24h from its reason split
feat(telemetry): preflight baseline counters (filter-diagnostics enga…
feat(preflight): required-tools preflight — deterministic availabilit…
spec: describe_tool check mode — in-band preflight (099)
7 remaining items
fix(099): positive operator grant, raw check arguments, glue-driven m…
fix(099): round 2 — parity exclusions scoped per wire surface
feat(mcp): describe_tool check mode — in-band preflight (spec 099) (#998
Dumbris commentedon Aug 17, 2026
@neylwalecki — v0.58.0 is out with the preflight stack, ready to test against your scheduled-job flow: https://github.com/smart-mcp-proxy/mcpproxy-go/releases/tag/v0.58.0
What's in it for your case:
mcpproxy tools preflight <server:tool>… [-o json] [--wait 10s] [--pin id=sha256/v2:…] [--read-only-only]— exit codes0ready ·10retryable ·11blocked ·12unknown id (worst class wins), so your cron wrapper branches without parsing JSON.POST /api/v1/preflight— the schema exactly as posted above (verdict is body data, HTTP 200 whenever the check executed; 400 validation / 503 can't-answer-honestly).describe_tool {check: true, tool_ids: […]}also landed (the thin in-band path you flagged for later — same evaluator, same codes, batch of 50, verdict-only), so an agent mid-session can gate its own plan too.server_initializingis strictly server-level.mcpproxy activity list --type preflightor--request-id <id>reconstructs any run.One heads-up from the release notes: the
describe_toolper-id error codeinvisiblewas retired (out-of-scope ids now reportnot_found) — a disclosure fix; only relevant if you ever branched on that string.Your six failure reasons all have dedicated codes and each ships with a sabotage E2E cell (we deliberately quarantine/disable/rug-pull/kill fixtures in CI and assert the exact code). If anything in the real flow surprises you — wrong reason, wrong retryability, anything ambiguous — that's exactly the feedback we want here.
neylwalecki commentedon Aug 17, 2026
Thanks for the ping — I tested the v0.58.0 macOS ARM64 release in an isolated scheduled-job fixture, and the preflight contract behaved as expected.
The CLI returned the exact branching signals we need: ready → 0, server_initializing/retryable → 10, missing_annotation and policy_filtered → 11, and not_found → 12. A mixed batch also respected the documented worst-class-wins rule. The REST endpoint returned HTTP 200 with a blocked verdict in the response body, and the corresponding preflight activity records were present.
The missing_annotation vs policy_filtered split is particularly useful for our automation because it distinguishes an upstream metadata fix from an intentionally unsafe tool. I also ran the focused preflight, CLI, sabotage-matrix, zero-upstream-I/O, and describe_tool check-mode tests from the v0.58.0 tag; they passed.
After that isolated validation, I updated the operational DMG installation from v0.57.0 to v0.58.0 and ran the preflight against the four stable finance tool IDs from the original scheduled-job case with --read-only-only. All four returned ready and the CLI exited 0. No upstream tool was invoked by the check.
One small documentation mismatch surfaced: specs/098-tools-preflight/quickstart.md points to internal/testdata/ctl-server.js, while the fixture in the v0.58.0 tag is at internal/server/testdata/preflight_fixture_server.js. Using the latter worked.
The implementation itself matches the cron/CI boundary we discussed, while the shared describe_tool check path covers the in-session case without adding another top-level tool. Thank you for carrying the original operational case through to a tested release.
Dumbris commentedon Aug 17, 2026
One more pattern worth knowing for your scheduled jobs, since it compounds the token savings: preflight + code execution / stored scripts.
Instead of the agent orchestrating N tool calls (N schemas + N intermediate results all flowing through model context), gate once and run the whole workflow server-side as one call:
Inside the script,
call_tool()/call_tools()(batched, parallel) do the multi-step work — intermediate payloads never enter model context, only the script's final value returns. With a stored script the code itself stays out of context too: the agent sends 40 bytes, not the program. For an N-step read-only pipeline that's roughly N×(schema+result) worth of context collapsed into one verdict block plus one result value.The two features also interlock on the safety side: as of v0.58.0 scripts are policy-gated through the same shared classification as preflight, so a mid-run quarantine flip fails the script with a structured refusal rather than a silent partial run — preflight gives you the pre-run answer, the gate guarantees the in-run one, and both name the same reason.
Docs: Required-Tools Preflight (composition section) + Code Execution.
docs(098): fix quickstart fixture path and sabotage recipes (#1000)
Dumbris commentedon Aug 17, 2026
Thanks for the thorough validation — running the tag's own sabotage/zero-I/O suites and the original four-tool case in production is exactly the closure this issue needed.
The quickstart mismatch you caught is fixed and merged in #1000: the doc now points at the fixture that actually ships (
internal/server/testdata/preflight_fixture_server.js) and the recipes were rewritten to be copy-paste runnable against its real knobs (FIXTURE_TOOLS_FILE/FIXTURE_INIT_DELAY_MS/FIXTURE_FAIL_FILE) — updated quickstart.With your production validation in, I'm closing this out as delivered (items 1 + 3, plus the in-band path). Item 2 (upstream-scoped retrieval) and the roadmap layers (readyz probe, tool lockfile, declared automation contracts) are tracked for future cycles — new issues welcome any time the scheduled-job flow surfaces something the taxonomy doesn't name cleanly.
docs(research): app.mcpproxy/required-tools extension payload spec + …
docs(specs): 102-schema-deferred plan hardening from verified review …
docs(specs): close cross-model review findings in the 102 task list
feat(102): self-healing direct calls (T019a, T059-T064)
feat(102): schema-deferred direct mode — complete (89/89 tasks) (#1063)
Dumbris commentedon Sep 2, 2026
Shipped across three specs.
retrieve_toolsnow returns afilter_diagnosticsblock reporting how many candidates matched before annotation gates and how many each gate removed, with a suggestion string.invalid_paramserrors landed with spec 085.describe_toolcheck mode landed as specs 098 and 099.Thanks for the report — the annotation-metadata failure mode you described is exactly what drove the diagnostics design. Closing; reopen if the shipped output does not cover your case.