bug(vmcp): Ready and /health stay OK while initialize can hang past client timeouts

[RaviTharuma]

Status

Fix PR: #6380

• Maintainer direction (bound initialize / document Ready vs health): bug(vmcp): Ready and /health stay OK while initialize can hang past client timeouts #6345 (comment)

• PR announcement: bug(vmcp): Ready and /health stay OK while initialize can hang past client timeouts #6345 (comment)

• Reply to maintainer: bug(vmcp): Ready and /health stay OK while initialize can hang past client timeouts #6345 (comment)

• PR cross-link comment: fix(vmcp): bound initialize to healthCheckTimeout #6380 (comment)

---

Bug description

VirtualMCPServer can report healthy while a client-facing initialize hangs long enough to look down.

On v0.43.0 we still see:

• CR Ready=True, status.discoveredBackends[].status=ready, circuit breaker closed, message=Healthy
• GET /health → 200 {"status":"ok"} (when probed in-cluster / via the proxy)
• POST /mcp initialize from a real client: one request timed out at 12s with 0 bytes; the next request to the same path returned 200 in ~3s
• The same aggregate on a prefixed path (/<name>/mcp) answered immediately

So “Ready” / /health / backend Healthy are not a reliable signal that a new MCP session will succeed within a normal gateway/client timeout.

#5861 (slow backend coin-flips the tenant) was closed as fixed by #6162 (“skip known-bad backends when opening sessions”). This looks like a leftover of the same class: health is not the same as “session open is fast and succeeds”.

Steps to reproduce

1. Run a VirtualMCPServer with several backends (we used 5), exposed through an HTTP gateway.
2. Confirm kubectl get virtualmcpserver Ready and every discoveredBackends[].status=ready.
3. From outside the cluster, POST initialize with a 12s client timeout, several times in a row, including after a period of idle.

We are not claiming a deterministic 50% fail rate anymore (that was #5861 on v0.34.0). We are claiming Ready/health can be green while initialize still sometimes exceeds 12s or returns nothing.

Expected behavior

• /health and the Ready condition should fail (or go Degraded) if a new session cannot initialize within the configured timeout.
• Alternatively, document that Ready only means “process up + last backend probe ok”, and ship a separate SessionReady / latency SLO.

Actual behavior

Ready + Healthy + /health 200 while a client initialize can still time out. A retry a few seconds later succeeds. Easy to page the wrong layer (gateway vs vMCP vs backend).

Environment (if relevant)

• OS/version: Talos Linux, Kubernetes, Cilium Gateway API in front of the vMCP Service
• ToolHive version: operator + vmcp v0.43.0

Additional context

Related: #5861 (closed, #6162), #6339 (degraded for unadvertised capabilities — inverse: status too pessimistic), #6276 (GET /metrics returns MCP 406; /health is the one unauthenticated JSON endpoint that does answer).

If the remaining hang is “first session after idle aggregates all backends synchronously”, a bounded initialize budget + best_effort default would match what #5861 asked for.

No credentials or private hostnames in this report.

[reyortiz3]

Thanks for the thorough report, @RaviTharuma, and for tying this back to #5861/#6162 — that context is useful. We have read this and it is noted.

It is not on our roadmap right now, but it is a good candidate for a community contribution — Ready currently only reflects whether a backend answers ListCapabilities within the health-check timeout, which is a different operation from a new client session's initialize handshake, and closing that gap (or documenting the distinction with a separate SessionReady signal) would be welcome. If you or anyone else would like to pick it up, open a draft PR or comment here and we will help align on direction.

Thanks again for taking the time to file it.

[RaviTharuma]

Opened #6380. initialize is now bounded by healthCheckTimeout (default 10s) so it cannot hang past typical gateway timeouts while Ready and /health stay OK. Those three signals are documented; Ready//health are not flipped on initialize failure (that would flap).

[RaviTharuma]

@reyortiz3 following your direction: opened #6380. initialize is bounded by healthCheckTimeout (default 10s). Ready / /health are documented as different signals and are not flipped on initialize failure.

Cross-links:

• Issue: bug(vmcp): Ready and /health stay OK while initialize can hang past client timeouts #6345
• PR: fix(vmcp): bound initialize to healthCheckTimeout #6380
• Your comment: bug(vmcp): Ready and /health stay OK while initialize can hang past client timeouts #6345 (comment)
• First PR note: bug(vmcp): Ready and /health stay OK while initialize can hang past client timeouts #6345 (comment)

[RaviTharuma]

Addressed in:

• fix(vmcp): bound initialize to healthCheckTimeout #6380 (OPEN)

Closing this issue because the fix is already filed as an upstream PR. The PR stays open for review; not merging from here.

[RaviTharuma]

Resolved via #6380