What happened
In pkg/scraper/client/resource/client.go, kubeletClient.getMetrics reads
the entire /metrics/resource response from a kubelet with:
_, err = io.Copy(buf, response.Body)
buf is a bytes.Buffer backed by a sync.Pool s
io.LimitReader or http.MaxBytesReader wrapping response.Body anywhere in
this path. The HTTP client's Timeout is inherited from the caller's
rest.Config and is commonly unset (0 = unlimited)n
how much gets buffered is the per-node scrape context timeout (default 10s),
which limits total bytes by bandwidth × time rather than an explicit size.
pkg/scraper/scraper.go launches one goroutine per node with no concurrency
cap, so multiple nodes are scraped in parallel usin.
Why this matters
If a kubelet returns a much larger response than ex
misbehaving metrics endpoint, or a compromised node), metrics-server will
buffer all of it in memory before doing any parsing or validation. Because
this happens per-node and in parallel, it's possibl
oversized response to push metrics-server's own memory usage well past
expected levels, risking an OOM-kill of the metrics-server pod and taking the
metrics API down for the whole cluster until it rec
Suggested fix
Bound the read independently of the peer, e.g. wrap the body before copying:
limited := io.LimitReader(response.Body, maxRes
n, err := io.Copy(buf, limited)
// treat n == maxResponseBytes as "possibly trut error
with maxResponseBytes set comfortably above the l
/metrics/resource payload. The buffer returned to the sync.Pool in the
deferred cleanup should also be capped so an oversi
large slice sitting in the pool.
Versions
- metrics-server: v0.8.0 (commit d66279c)
- Location:
pkg/scraper/client/resource/client.gos`
(io.Copy call, ~line 127 on this tag)
What happened
In
pkg/scraper/client/resource/client.go,kubeletClient.getMetricsreadsthe entire
/metrics/resourceresponse from a kubelet with:bufis abytes.Bufferbacked by async.Poolsio.LimitReaderorhttp.MaxBytesReaderwrappingresponse.Bodyanywhere inthis path. The HTTP client's
Timeoutis inherited from the caller'srest.Configand is commonly unset (0 = unlimited)nhow much gets buffered is the per-node scrape context timeout (default 10s),
which limits total bytes by bandwidth × time rather than an explicit size.
pkg/scraper/scraper.golaunches one goroutine per node with no concurrencycap, so multiple nodes are scraped in parallel usin.
Why this matters
If a kubelet returns a much larger response than ex
misbehaving metrics endpoint, or a compromised node), metrics-server will
buffer all of it in memory before doing any parsing or validation. Because
this happens per-node and in parallel, it's possibl
oversized response to push metrics-server's own memory usage well past
expected levels, risking an OOM-kill of the metrics-server pod and taking the
metrics API down for the whole cluster until it rec
Suggested fix
Bound the read independently of the peer, e.g. wrap the body before copying:
with
maxResponseBytesset comfortably above the l/metrics/resourcepayload. The buffer returned to thesync.Poolin thedeferred cleanup should also be capped so an oversi
large slice sitting in the pool.
Versions
pkg/scraper/client/resource/client.gos`(io.Copy call, ~line 127 on this tag)