Skip to content

Commit ca00071

Browse files
authored
fix(telemetry): flush a final heartbeat on graceful shutdown (#1037)
* fix(telemetry): flush a final heartbeat on graceful shutdown Telemetry counters are in-memory and reset only after an accepted (2xx) send, so everything recorded after the 5-minute first heartbeat was lost unless the process survived to the 24h tick — which most desktop sessions do not. Deferred follow-up from #1029. Stop() now performs ONE final heartbeat after joining the loop: - It reuses sendHeartbeat's send+reset path verbatim, so the counter contract is unchanged — a failed/timed-out flush drops nothing, an accepted flush cannot double-count with the next start. - It runs AFTER the loop is joined rather than from a second goroutine, so there is provably no concurrent sender and the flush cannot interleave with a tick. - It uses a fresh context hard-bounded to 4s (Runtime.Close cancels the app context before Stop, and a dead endpoint must never hang a quit). - It is skipped when telemetry is disabled by env/config, the build is non-semver, the user opted out, or nothing was recorded since the last accepted send (new CounterRegistry.HasPendingCounters). - Anonymous-id rotation is suppressed on this path: rotation rewrites the whole config file, and shutdown is the wrong moment for that. Also closes a pre-existing data race the flush widens: s.endpoint is rewritten under s.mu by NotifyConfigChanged but was read unlocked by the heartbeat, feedback, and opt-out-beacon paths. All four now read it through liveEndpoint(), matching liveConfig(). * fix(telemetry): don't let the shutdown flush spend one-shot state Cross-model review round 1 (opencode / gpt-5.6-sol) found three real defects in the graceful-shutdown flush. 1. A failed flush destroyed the installer attribution. resolveLaunchSource clears installer_heartbeat_pending at BUILD time, before the send is known to have landed — a deliberate trade for the long-running loop, but it inverts on the flush: the process is exiting, so an offline or timed-out flush spent a one-shot that previously survived into the next run. The flush now leaves the flag armed and reports the runtime-detected source; the next start still claims "installer" exactly once. Generalized the rotate flag into consumeOneShots, which now gates both the 365-day anonymous-id rotation and the installer flag (send/buildHeartbeat* renamed accordingly). 2. TestFlushOnStopNoDoubleSendWithPeriodicTick was flaky — reproduced at -count=300 (got 4, want 2). Its premise was wrong: a tick aborted after the endpoint accepted it leaves counters pending, so the flush re-sends them. Delivery is at-least-once (undecidable client-side), which is the deliberate trade against dropping the window entirely. Split into two deterministic tests: the flush skips after an ACCEPTED tick, and joins + re-sends an unconfirmed one. Documented the semantics in code and docs. 3. TestFlushOnStopSkippedWhenOptedOut raced the opt-out beacon, which POSTs to the same /heartbeat ingest path (its comment claimed /opt-out). The recorder now classifies beacons by their `event` field and the test waits for the beacon before asserting no usage heartbeat followed. Rejected as pre-existing/undecidable, with rebuttals: heartbeat-URL validateTelemetryURL pinning (pre-existing sink, local config value), buildHeartbeat not bounded by flushTimeout (the joined tick already had the same exposure), and the opt-out latch race (the flush's telemetryEnabledLive plus the pre-transmit optedOut re-check already close the described window). * fix(telemetry): read the anonymous id under s.mu in the opt-out beacon Cross-model review round 2 (opencode / gpt-5.6-sol). One genuine finding: the second half of this PR's own s.endpoint drive-by was left undone. EmitOptOutBeacon and SendOptOutBeacon both gate on s.config.GetAnonymousID() read WITHOUT s.mu, two lines above the endpoint read that now takes it. NotifyConfigChanged swaps s.config under s.mu, and GetAnonymousID dereferences cfg.Telemetry — a pointer ensureAnonymousIDOnce / advanceUpgradeFunnelOnce also install under s.mu — so snapshotting the pointer would not have been enough either; the whole read has to be inside the critical section, exactly as telemetryEnabledLive documents. NotifyConfigChanged itself launches the beacon goroutine, so the writer and the reader are one call apart. Added liveAnonymousID() alongside liveConfig()/liveEndpoint() and routed both gates through it. TestOptOutBeaconReadsAnonymousIDUnderLock reproduces the race first (the detector flags both sites against optout.go:215) and passes after. No unlocked s.config read remains outside a locked region. Also re-raised and rejected again: the opt-out TOCTOU between the pre-transmit optedOut check and client.Do. It is pre-existing on the periodic sender and unchanged in character here, and the proposed fix — an emission mutex held across the HTTP send — would make NotifyConfigChanged block the config-save path on up to 10s of network I/O, breaking the "never blocks the caller" invariant that goroutine exists to uphold.
1 parent 89af5c5 commit ca00071

8 files changed

Lines changed: 809 additions & 15 deletions

File tree

‎docs/features/telemetry.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,16 @@ The `docker_cli_source` field is likewise a **fixed enum** (`path`, `bundled`, `
4949

5050
Docker isolation failures surface in `error_code_counts_24h` via three stable diagnostic codes (schema v5): `MCPX_DOCKER_CLI_NOT_FOUND` (isolation requested but the `docker` binary is unresolved — issue #696), `MCPX_DOCKER_EXEC_NOT_FOUND` (the image lacks the interpreter the server needs, e.g. `uvx` missing in `python:3.11`), and `MCPX_DOCKER_OCI_RUNTIME` (OCI runtime / architecture-mismatch failures).
5151

52+
## When heartbeats are sent
53+
54+
A heartbeat goes out 5 minutes after start (so short-lived processes stay quiet), then once every 24 hours, and **once more on graceful shutdown**.
55+
56+
The shutdown heartbeat exists because the windowed counters live in memory and are cleared only after the endpoint accepts a send. Without it, everything an install did after its 5-minute heartbeat was discarded whenever the process exited before the 24-hour tick — which is most desktop sessions. It is skipped when telemetry is disabled or opted out, and when nothing has been recorded since the last accepted heartbeat. It is hard-bounded to a few seconds, so an unreachable endpoint can never delay quitting, and because counters are cleared only on an accepted send, a failed shutdown flush simply leaves the counts for the next run rather than dropping them.
57+
58+
Delivery is therefore **at-least-once**, not exactly-once. If a send is cut off after the endpoint accepted it but before mcpproxy sees the response, the counts stay pending locally and are re-sent — a lost response is indistinguishable from a lost request. Receivers should treat a heartbeat as idempotent per `(anonymous_id, timestamp)`. Retrying is the deliberate choice: the alternative is silently discarding the window, which is the problem the shutdown heartbeat was added to solve.
59+
60+
Two housekeeping steps are deliberately *not* performed by the shutdown heartbeat — the 365-day anonymous-ID rotation and the one-shot `installer` launch-source attribution. Both are consumed when the payload is built rather than when it is accepted, so running them for a send that may never land would spend state a normal run still has. The next start performs both.
61+
5262
## What is NOT collected
5363

5464
The following is **never** collected:

‎internal/runtime/runtime.go‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -776,6 +776,13 @@ func (r *Runtime) Close() error {
776776
// ActivityService.Stop above — terminally stops the service so a Start
777777
// goroutine not yet scheduled (lifecycle.go launches it via `go`) becomes
778778
// a no-op instead of writing after this point.
779+
//
780+
// Stop also performs the graceful-shutdown heartbeat flush — one bounded
781+
// (4s) final send of the counters recorded since the last accepted
782+
// heartbeat, which would otherwise die with the process. It runs after the
783+
// loop is joined, so it is the only sender at that moment, and it happens
784+
// HERE — while the BBolt handle is still open and before the marker
785+
// resolves below — because its buildHeartbeat writes funnel activity.
779786
if r.telemetryService != nil {
780787
r.telemetryService.Stop()
781788
}

‎internal/telemetry/activation_test.go‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -337,22 +337,36 @@ func TestInstallerPending_ClearedAfterFirstHeartbeat(t *testing.T) {
337337
activationDB: db,
338338
}
339339

340-
// First call → "installer" and the pending flag must be cleared.
341-
ls1 := s.resolveLaunchSource()
340+
// The shutdown flush must NOT consume the one-shot: it builds a payload that
341+
// may never be accepted, so a non-consuming resolve has to leave the flag
342+
// armed for the next run and report the runtime-detected source instead.
343+
if ls := s.resolveLaunchSource(false); ls == LaunchSourceInstaller {
344+
t.Fatal("non-consuming resolveLaunchSource claimed installer; the shutdown flush must not spend the one-shot")
345+
}
346+
pendingAfterFlush, err := store.IsInstallerPending(db)
347+
if err != nil {
348+
t.Fatalf("IsInstallerPending after non-consuming resolve: %v", err)
349+
}
350+
if !pendingAfterFlush {
351+
t.Fatal("non-consuming resolveLaunchSource cleared installer_heartbeat_pending; a failed shutdown flush would destroy the attribution")
352+
}
353+
354+
// First consuming call → "installer" and the pending flag must be cleared.
355+
ls1 := s.resolveLaunchSource(true)
342356
if ls1 != LaunchSourceInstaller {
343357
t.Fatalf("first resolveLaunchSource = %q, want %q", ls1, LaunchSourceInstaller)
344358
}
345359
pending, err := store.IsInstallerPending(db)
346360
if err != nil {
347-
t.Fatalf("IsInstallerPending after first call: %v", err)
361+
t.Fatalf("IsInstallerPending after first consuming call: %v", err)
348362
}
349363
if pending {
350364
t.Fatalf("installer_heartbeat_pending should be cleared after first heartbeat")
351365
}
352366

353367
// Second call → no longer installer; should return the runtime
354368
// detector's value (whatever it is in the test env), not installer.
355-
ls2 := s.resolveLaunchSource()
369+
ls2 := s.resolveLaunchSource(true)
356370
if ls2 == LaunchSourceInstaller {
357371
t.Fatalf("second resolveLaunchSource still returned installer — one-shot broken")
358372
}

‎internal/telemetry/feedback.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ func (s *Service) SubmitFeedback(ctx context.Context, req *FeedbackRequest) (*Fe
133133
return nil, fmt.Errorf("failed to marshal feedback: %w", err)
134134
}
135135

136-
url := s.endpoint + "/feedback"
136+
url := s.liveEndpoint() + "/feedback"
137137
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(data))
138138
if err != nil {
139139
return nil, fmt.Errorf("failed to create feedback request: %w", err)

‎internal/telemetry/optout.go‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ func (s *Service) EmitOptOutBeacon(ctx context.Context) bool {
7979
if disabled, _ := IsDisabledByEnv(); disabled {
8080
return false
8181
}
82-
if s.config.GetAnonymousID() == "" {
82+
if s.liveAnonymousID() == "" {
8383
return false
8484
}
8585
if err := s.SendOptOutBeacon(ctx); err != nil {
@@ -98,7 +98,7 @@ func (s *Service) EmitOptOutBeacon(ctx context.Context) bool {
9898
// caller-supplied URL, so this never sends to an arbitrary, request-derived
9999
// host.
100100
func (s *Service) SendOptOutBeacon(ctx context.Context) error {
101-
anonID := s.config.GetAnonymousID()
101+
anonID := s.liveAnonymousID()
102102
if anonID == "" {
103103
// Nothing to dedup on — never send an identity-less beacon.
104104
return errors.New("opt-out beacon skipped: no anonymous_id")
@@ -122,7 +122,7 @@ func (s *Service) SendOptOutBeacon(ctx context.Context) error {
122122
// require a host, and issue the request against the re-serialized URL. This
123123
// mirrors validateRegistryURL and guarantees a malformed/non-http endpoint
124124
// can never aim the beacon at, e.g., file:// or a schemeless host.
125-
beaconURL, err := validateTelemetryURL(strings.TrimRight(s.endpoint, "/") + "/heartbeat")
125+
beaconURL, err := validateTelemetryURL(strings.TrimRight(s.liveEndpoint(), "/") + "/heartbeat")
126126
if err != nil {
127127
return err
128128
}

‎internal/telemetry/registry.go‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -490,6 +490,50 @@ func (r *CounterRegistry) Snapshot() RegistrySnapshot {
490490
return snap
491491
}
492492

493+
// HasPendingCounters reports whether anything has been recorded since the last
494+
// Reset — i.e. whether a heartbeat would carry usage data that has not yet been
495+
// accepted by the endpoint.
496+
//
497+
// The counter set inspected here is EXACTLY the set Reset zeroes: those are the
498+
// counters whose window is "since the last accepted send", so they are the ones
499+
// that would be lost if the process exited without transmitting. Lifetime
500+
// tallies that Reset deliberately preserves (anonymityViolations) are excluded —
501+
// they are never transmitted and must not, on their own, justify a send.
502+
//
503+
// Used by the graceful-shutdown flush to skip a final heartbeat that would carry
504+
// no new information.
505+
func (r *CounterRegistry) HasPendingCounters() bool {
506+
if r == nil {
507+
return false
508+
}
509+
for i := range r.surfaceCounts {
510+
if r.surfaceCounts[i].Load() > 0 {
511+
return true
512+
}
513+
}
514+
if r.upstreamTotal.Load() > 0 {
515+
return true
516+
}
517+
518+
r.mu.RLock()
519+
defer r.mu.RUnlock()
520+
521+
if len(r.builtinCalls) > 0 || len(r.restEndpoints) > 0 ||
522+
len(r.errorCategories) > 0 || len(r.doctorChecks) > 0 {
523+
return true
524+
}
525+
if r.tpaScansCompleted > 0 || r.tpaScansFailed > 0 || r.tpaScansWithFindings > 0 ||
526+
r.tpaToolChangeGateScans > 0 || r.tpaPromptScans > 0 {
527+
return true
528+
}
529+
for _, v := range r.tpaFindings {
530+
if v > 0 {
531+
return true
532+
}
533+
}
534+
return false
535+
}
536+
493537
// Reset zeros all counters. Called only after a successful heartbeat send.
494538
//
495539
// Deliberate, registry-wide trade-off: an event recorded between Snapshot()

0 commit comments

Comments
 (0)