Skip to content

MITM rate-limit denials (CONNECT/forward) are silent — no log, no metric, no request_logs row #381

Description

@jhgaylor

Summary

When the MITM proxy denies a CONNECT (or absolute-form forward) for
rate-limiting, it writes the 429 and returns without logging or counting
anything
. TLS handshake failures on the same path do log
(p.logger.Warn("mitm TLS handshake failed", …)), so a rate-limit denial is
strictly less observable than a handshake error — even though a denial can take
down every client behind a shared ingress IP (see #380).

Where

  • internal/mitm/connect.gohandleConnect:
    if d := p.rateLimit.Check(ratelimit.TierAuth, mitmIPKey(r)); !d.Allow {
        ratelimit.WriteDenial(w, d, "Too many CONNECT attempts")
        return   // no logger call, no metric
    }
  • internal/mitm/forward.gohandleForward has the identical silent
    WriteDenial(... "Too many proxy requests").

Impact

During a live incident, the proxy was 429-ing authenticated CONNECTs
fleet-wide (#380). From the broker side the outage was invisible: no WARN,
and denied CONNECTs never reach forwardHandler, so there is no
request_logs row
for the target host either. The only evidence was in the
clients (git's "Could not connect to server") and the absence of forward
rows. Diagnosis required reproducing the 429 by hand.

Suggested fix

At each MITM rate-limit denial, log at WARN with the tier, the key (or a
redacted form), and Retry-After, matching the existing handshake-failure log.
A counter would be even better — e.g. agentvault_mitm_ratelimit_denied_total
by tier/reason — which also slots into the /metrics endpoint proposed in #329.
Denials being silent is the specific gap; #329 is the general observability
ask.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions