Summary
When the MITM proxy denies a CONNECT (or absolute-form forward) for
rate-limiting, it writes the 429 and returns without logging or counting
anything. TLS handshake failures on the same path do log
(p.logger.Warn("mitm TLS handshake failed", …)), so a rate-limit denial is
strictly less observable than a handshake error — even though a denial can take
down every client behind a shared ingress IP (see #380).
Where
internal/mitm/connect.go — handleConnect:
if d := p.rateLimit.Check(ratelimit.TierAuth, mitmIPKey(r)); !d.Allow {
ratelimit.WriteDenial(w, d, "Too many CONNECT attempts")
return // no logger call, no metric
}
internal/mitm/forward.go — handleForward has the identical silent
WriteDenial(... "Too many proxy requests").
Impact
During a live incident, the proxy was 429-ing authenticated CONNECTs
fleet-wide (#380). From the broker side the outage was invisible: no WARN,
and denied CONNECTs never reach forwardHandler, so there is no
request_logs row for the target host either. The only evidence was in the
clients (git's "Could not connect to server") and the absence of forward
rows. Diagnosis required reproducing the 429 by hand.
Suggested fix
At each MITM rate-limit denial, log at WARN with the tier, the key (or a
redacted form), and Retry-After, matching the existing handshake-failure log.
A counter would be even better — e.g. agentvault_mitm_ratelimit_denied_total
by tier/reason — which also slots into the /metrics endpoint proposed in #329.
Denials being silent is the specific gap; #329 is the general observability
ask.
Summary
When the MITM proxy denies a
CONNECT(or absolute-form forward) forrate-limiting, it writes the 429 and returns without logging or counting
anything. TLS handshake failures on the same path do log
(
p.logger.Warn("mitm TLS handshake failed", …)), so a rate-limit denial isstrictly less observable than a handshake error — even though a denial can take
down every client behind a shared ingress IP (see #380).
Where
internal/mitm/connect.go—handleConnect:internal/mitm/forward.go—handleForwardhas the identical silentWriteDenial(... "Too many proxy requests").Impact
During a live incident, the proxy was 429-ing authenticated
CONNECTsfleet-wide (#380). From the broker side the outage was invisible: no WARN,
and denied
CONNECTs never reachforwardHandler, so there is norequest_logsrow for the target host either. The only evidence was in theclients (
git's "Could not connect to server") and the absence of forwardrows. Diagnosis required reproducing the 429 by hand.
Suggested fix
At each MITM rate-limit denial, log at WARN with the tier, the key (or a
redacted form), and
Retry-After, matching the existing handshake-failure log.A counter would be even better — e.g.
agentvault_mitm_ratelimit_denied_totalby tier/reason — which also slots into the
/metricsendpoint proposed in #329.Denials being silent is the specific gap; #329 is the general observability
ask.