Repository navigation
Commit ca00071
authored
fix(telemetry): flush a final heartbeat on graceful shutdown (#1037)
* fix(telemetry): flush a final heartbeat on graceful shutdown
Telemetry counters are in-memory and reset only after an accepted (2xx)
send, so everything recorded after the 5-minute first heartbeat was lost
unless the process survived to the 24h tick — which most desktop sessions
do not. Deferred follow-up from #1029.
Stop() now performs ONE final heartbeat after joining the loop:
- It reuses sendHeartbeat's send+reset path verbatim, so the counter
contract is unchanged — a failed/timed-out flush drops nothing, an
accepted flush cannot double-count with the next start.
- It runs AFTER the loop is joined rather than from a second goroutine,
so there is provably no concurrent sender and the flush cannot
interleave with a tick.
- It uses a fresh context hard-bounded to 4s (Runtime.Close cancels the
app context before Stop, and a dead endpoint must never hang a quit).
- It is skipped when telemetry is disabled by env/config, the build is
non-semver, the user opted out, or nothing was recorded since the last
accepted send (new CounterRegistry.HasPendingCounters).
- Anonymous-id rotation is suppressed on this path: rotation rewrites the
whole config file, and shutdown is the wrong moment for that.
Also closes a pre-existing data race the flush widens: s.endpoint is
rewritten under s.mu by NotifyConfigChanged but was read unlocked by the
heartbeat, feedback, and opt-out-beacon paths. All four now read it
through liveEndpoint(), matching liveConfig().
* fix(telemetry): don't let the shutdown flush spend one-shot state
Cross-model review round 1 (opencode / gpt-5.6-sol) found three real defects
in the graceful-shutdown flush.
1. A failed flush destroyed the installer attribution. resolveLaunchSource
clears installer_heartbeat_pending at BUILD time, before the send is known
to have landed — a deliberate trade for the long-running loop, but it
inverts on the flush: the process is exiting, so an offline or timed-out
flush spent a one-shot that previously survived into the next run. The
flush now leaves the flag armed and reports the runtime-detected source;
the next start still claims "installer" exactly once. Generalized the
rotate flag into consumeOneShots, which now gates both the 365-day
anonymous-id rotation and the installer flag (send/buildHeartbeat*
renamed accordingly).
2. TestFlushOnStopNoDoubleSendWithPeriodicTick was flaky — reproduced at
-count=300 (got 4, want 2). Its premise was wrong: a tick aborted after
the endpoint accepted it leaves counters pending, so the flush re-sends
them. Delivery is at-least-once (undecidable client-side), which is the
deliberate trade against dropping the window entirely. Split into two
deterministic tests: the flush skips after an ACCEPTED tick, and joins +
re-sends an unconfirmed one. Documented the semantics in code and docs.
3. TestFlushOnStopSkippedWhenOptedOut raced the opt-out beacon, which POSTs
to the same /heartbeat ingest path (its comment claimed /opt-out). The
recorder now classifies beacons by their `event` field and the test waits
for the beacon before asserting no usage heartbeat followed.
Rejected as pre-existing/undecidable, with rebuttals: heartbeat-URL
validateTelemetryURL pinning (pre-existing sink, local config value),
buildHeartbeat not bounded by flushTimeout (the joined tick already had the
same exposure), and the opt-out latch race (the flush's telemetryEnabledLive
plus the pre-transmit optedOut re-check already close the described window).
* fix(telemetry): read the anonymous id under s.mu in the opt-out beacon
Cross-model review round 2 (opencode / gpt-5.6-sol). One genuine finding: the
second half of this PR's own s.endpoint drive-by was left undone.
EmitOptOutBeacon and SendOptOutBeacon both gate on s.config.GetAnonymousID()
read WITHOUT s.mu, two lines above the endpoint read that now takes it.
NotifyConfigChanged swaps s.config under s.mu, and GetAnonymousID dereferences
cfg.Telemetry — a pointer ensureAnonymousIDOnce / advanceUpgradeFunnelOnce also
install under s.mu — so snapshotting the pointer would not have been enough
either; the whole read has to be inside the critical section, exactly as
telemetryEnabledLive documents. NotifyConfigChanged itself launches the beacon
goroutine, so the writer and the reader are one call apart.
Added liveAnonymousID() alongside liveConfig()/liveEndpoint() and routed both
gates through it. TestOptOutBeaconReadsAnonymousIDUnderLock reproduces the
race first (the detector flags both sites against optout.go:215) and passes
after. No unlocked s.config read remains outside a locked region.
Also re-raised and rejected again: the opt-out TOCTOU between the pre-transmit
optedOut check and client.Do. It is pre-existing on the periodic sender and
unchanged in character here, and the proposed fix — an emission mutex held
across the HTTP send — would make NotifyConfigChanged block the config-save
path on up to 10s of network I/O, breaking the "never blocks the caller"
invariant that goroutine exists to uphold.1 parent 89af5c5 commit ca00071
8 files changed
Lines changed: 809 additions & 15 deletions
File tree
- docs/features
- internal
- runtime
- telemetry
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
52 | 62 | | |
53 | 63 | | |
54 | 64 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
776 | 776 | | |
777 | 777 | | |
778 | 778 | | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
779 | 786 | | |
780 | 787 | | |
781 | 788 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
337 | 337 | | |
338 | 338 | | |
339 | 339 | | |
340 | | - | |
341 | | - | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
342 | 356 | | |
343 | 357 | | |
344 | 358 | | |
345 | 359 | | |
346 | 360 | | |
347 | | - | |
| 361 | + | |
348 | 362 | | |
349 | 363 | | |
350 | 364 | | |
351 | 365 | | |
352 | 366 | | |
353 | 367 | | |
354 | 368 | | |
355 | | - | |
| 369 | + | |
356 | 370 | | |
357 | 371 | | |
358 | 372 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
136 | | - | |
| 136 | + | |
137 | 137 | | |
138 | 138 | | |
139 | 139 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
82 | | - | |
| 82 | + | |
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
101 | | - | |
| 101 | + | |
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
| |||
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
125 | | - | |
| 125 | + | |
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
490 | 490 | | |
491 | 491 | | |
492 | 492 | | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
493 | 537 | | |
494 | 538 | | |
495 | 539 | | |
| |||
0 commit comments