Skip to content

fix(observability): label HTTP metrics by route pattern, not raw path - #1791

Open
rudycelekli wants to merge 1 commit into
smart-mcp-proxy:mainfrom
rudycelekli:fix/metrics-route-pattern-label-20261025
Open

rudycelekli wants to merge 1 commit into
smart-mcp-proxy:mainfrom
rudycelekli:fix/metrics-route-pattern-label-20261025

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

Bug

With observability.metrics.enabled, MetricsManager.HTTPMiddleware is the first middleware on the REST API chi router (setupRoutes), ahead of apiKeyAuthMiddleware. It labels mcpproxy_http_requests_total and mcpproxy_http_request_duration_seconds with r.URL.Path.

So every distinct path creates a new counter series and a new histogram series (11 buckets plus sum and count), and they are never removed. That includes unauthenticated requests that end in 401 or 404, and every server name in /api/v1/servers/{name}/….

Trigger

With metrics enabled, any client that can reach the listener sends requests to distinct paths, for example GET /api/v1/<random> in a loop. A web page open in the user's browser can do this against 127.0.0.1 with fetch(..., {mode: "no-cors"}), and no API key is needed. The daemon's memory and the size of every /metrics scrape then grow without bound.

Fix

Label requests with the chi route pattern they matched (/api/v1/servers/{name}). A request that matched no route gets one fixed unmatched label. Outside a chi router, which no production caller uses, the raw path is kept. The label name (path) and the other labels are unchanged, so the Grafana dashboard, which sums by status, keeps working.

Test

TestHTTPMiddleware_PathLabelIsRoutePattern: a chi router with one /api/v1/servers/{name} route. It receives requests for two server names and two unknown paths. It must end up with exactly two series: the route pattern with count 2, and unmatched with count 2.

Commands run

  • CGO_ENABLED=0 go test -count=1 -run 'Test' ./internal/observability/: ok
  • On current main, and again with the fix reverted, the new test fails with expected: 2, actual: 4 series.
  • go vet ./internal/observability/: ok

The Prometheus HTTP middleware is the first middleware on the REST API
router, ahead of authentication, and it used r.URL.Path as the path label.
Every distinct path any caller sends (/api/v1/<random>, or a server name in
/api/v1/servers/{name}/...) therefore created a new counter and histogram
series that is never freed, so with metrics enabled an unauthenticated
client could grow the daemon's memory and the /metrics output without
bound.

Use the chi route pattern the request matched, and a single "unmatched"
label when no route matched.
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 75.00000% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/observability/metrics.go 75.00% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants