Repository navigation
fix(observability): label HTTP metrics by route pattern, not raw path - #1791
Open
rudycelekli wants to merge 1 commit into
Open
rudycelekli wants to merge 1 commit into
rudycelekli wants to merge 1 commit into
Conversation
The Prometheus HTTP middleware is the first middleware on the REST API
router, ahead of authentication, and it used r.URL.Path as the path label.
Every distinct path any caller sends (/api/v1/<random>, or a server name in
/api/v1/servers/{name}/...) therefore created a new counter and histogram
series that is never freed, so with metrics enabled an unauthenticated
client could grow the daemon's memory and the /metrics output without
bound.
Use the chi route pattern the request matched, and a single "unmatched"
label when no route matched.
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
With
observability.metrics.enabled,MetricsManager.HTTPMiddlewareis the first middleware on the REST API chi router (setupRoutes), ahead ofapiKeyAuthMiddleware. It labelsmcpproxy_http_requests_totalandmcpproxy_http_request_duration_secondswithr.URL.Path.So every distinct path creates a new counter series and a new histogram series (11 buckets plus sum and count), and they are never removed. That includes unauthenticated requests that end in 401 or 404, and every server name in
/api/v1/servers/{name}/….Trigger
With metrics enabled, any client that can reach the listener sends requests to distinct paths, for example
GET /api/v1/<random>in a loop. A web page open in the user's browser can do this against127.0.0.1withfetch(..., {mode: "no-cors"}), and no API key is needed. The daemon's memory and the size of every/metricsscrape then grow without bound.Fix
Label requests with the chi route pattern they matched (
/api/v1/servers/{name}). A request that matched no route gets one fixedunmatchedlabel. Outside a chi router, which no production caller uses, the raw path is kept. The label name (path) and the other labels are unchanged, so the Grafana dashboard, which sums bystatus, keeps working.Test
TestHTTPMiddleware_PathLabelIsRoutePattern: a chi router with one/api/v1/servers/{name}route. It receives requests for two server names and two unknown paths. It must end up with exactly two series: the route pattern with count 2, andunmatchedwith count 2.Commands run
CGO_ENABLED=0 go test -count=1 -run 'Test' ./internal/observability/: okmain, and again with the fix reverted, the new test fails withexpected: 2, actual: 4series.go vet ./internal/observability/: ok