Repository navigation
Commit 5e4698a
authored
feat(telemetry): heartbeat v9 — TPA gate/prompt scan counters + trust_mode distribution (#1029)
* feat(telemetry): heartbeat v9 — TPA gate/prompt scan counters + trust_mode distribution
The v8 tpa_scanner counters only see scan JOBS, which most installs never
start. The two TPA detection paths that run synchronously for ordinary users
emitted nothing at all, so the fleet read as "the scanner never runs":
- the trust_mode:scan tool-change gate (runtime.scanChangeIsClean)
- the aggregated-prompt poisoning filter (server.scanAggregatedPrompts)
Both now increment a delta counter in the tpa_scanner sub-object
(tool_change_gate_scans / prompt_scans) via the telemetry counter registry,
with the same window and reset-after-accepted-send semantics as every other
counter. Counts of invocations only — never the server, the tool/prompt, or
the verdict. The sub-object is still omitted when all counters (v8 and v9)
are zero.
Adds trust_mode_distribution: a state field counting configured servers per
EffectiveTrustMode(), keyed by the fixed auto|scan|manual enum and recomputed
from the live config on each heartbeat. It is the denominator the gate counter
needs (only "scan" servers can produce a gate scan) and the first fleet-wide
view of trust-tier adoption.
schema_version bumped to 9. The anonymity scanner widens the tpa_scanner key
whitelist and gains rule trust_mode_field_invalid so a producer-side
regression leaking a server name as a histogram key blocks the send.
* test(telemetry): note the v9 additions in the REST payload contract test
* fix(telemetry): build the heartbeat from one locked config snapshot
Cross-model review of the v9 payload flagged the trust_mode_distribution
builder reading s.config unsynchronized. NotifyConfigChanged replaces that
pointer wholesale under s.mu on every live config reload (REST apply, disk
watcher), so the read is a genuine data race — and it was not new: eight
sibling reads in buildHeartbeat, plus maybeRotateAnonymousID and
persistConfig, had the same exposure. trust_mode_distribution just made it
visible by walking cfg.Servers.
buildHeartbeat now resolves the live config exactly once through a new
liveConfig() accessor (lock, read, release — the mutex is non-reentrant and
preflightSink takes it, so it must not be held across the build) and every
downstream read uses that snapshot. maybeRotateAnonymousID and persistConfig
take the config as a parameter so the rotation mutates and persists the same
pointer the payload reported, instead of whatever s.config points at by then.
Adds a regression test that hammers NotifyConfigChanged against concurrent
buildHeartbeat calls; under -race it fails on the pre-fix code and also
asserts the histogram is never torn across a swap (all three fixed keys, two
servers total).
* fix(telemetry): route advanceUpgradeFunnel through the locked config accessor
advanceUpgradeFunnel runs on the heartbeat loop right after a 2xx send and
mutates config.Telemetry.LastReportedVersion, but it still read s.config
directly -- the same race with the NotifyConfigChanged pointer swap that
buildHeartbeat was just fixed for. Route it through liveConfig() so the whole
heartbeat build-and-send path reads the live config through one accessor.
* fix(telemetry): never persist a config the daemon has already swapped out
Round 2 of the cross-model review caught a regression in round 1's fix. The
heartbeat now works from a config SNAPSHOT, and persistConfig writes the WHOLE
config file — so an in-flight heartbeat could save its stale snapshot over a
config the user had just applied: snapshot A, user applies B (already on disk),
the in-flight anonymous-ID rotation saves A, B is silently rolled back.
persistConfig now writes only while the config it was handed is still s.config,
with the liveness check and the write under one s.mu hold so the swap cannot
slip between them. Skipping is safe and self-healing: every caller's mutation is
idempotent, so the next heartbeat re-evaluates it against the live config and
persists then. advanceUpgradeFunnel drops its open-coded SaveConfig and goes
through the same guarded path.
Two regression tests, both verified to fail with the guard disabled: a direct
persistConfig-after-swap case (asserting the guard is a liveness check, not a
blanket refusal — the live config still writes) and the end-to-end rotation
case that reaches persistConfig through maybeRotateAnonymousID.
* fix(telemetry): make the rotation and the upgrade cursor swap-safe
Round 3 of the cross-model review found the round-2 guard was correct but that
"skipping the write is safe because the mutation is idempotent" was not true for
two of its callers. persistConfig now reports whether it wrote, and both
callers act on that:
- Anonymous-ID rotation is all-or-nothing. The caller transmits the snapshot's
anonymous_id in the payload it is building, so an id that was never written
to disk must never leave the machine: one annual rotation would otherwise
surface as TWO identities (this heartbeat's unpersisted id, then the id the
next heartbeat rotates the live config to) and fragment the install's
telemetry continuity. On a skipped write the snapshot is restored.
- The upgrade-funnel cursor is not self-healing. Leaving last_reported_version
unadvanced makes the next heartbeat report the same previous_version again,
double-counting one upgrade, so the advance is redone against the new live
config when the guarded write was skipped.
persistConfig also now distinguishes "nothing to persist" from "write skipped":
a service with no cfgPath (in-memory/CLI use) has no on-disk state for the
mutation to be inconsistent with, so it reports success and callers keep their
mutation. Without that split the rotation rolled itself back in every test and
CLI service — caught by TestIDRotatesAfter365Days.
The residual clobber window the review also raised is documented on
persistConfig rather than papered over: the daemon's config writers save the
file BEFORE calling NotifyConfigChanged, so in that gap the old pointer still
looks live. Closing it needs single-writer ownership of the config file, a
config-layer change out of scope here; the guard narrows the exposure, it does
not eliminate it.
Tests: rotation rollback and the funnel end-state after a swap, both verified
to fail against the unfixed code.
* fix(telemetry): make the anonymous-ID rotation atomic
Round 4 of the cross-model review found the rotation rollback was itself not
concurrency-safe. BuildPayload is exported and served from an HTTP handler
(internal/httpapi), so a request can build a heartbeat concurrently with the
heartbeat loop and two rotations can race on the same config. Interleaved, one
captures the OTHER's freshly generated id as its "previous" value and restores
that on rollback — leaving a never-persisted id in the config the payload then
transmits, which is exactly the identity fragmentation the rollback was added
to prevent.
maybeRotateAnonymousID now holds s.mu across its whole check → generate →
mutate → persist-or-rollback sequence, so the rotation is atomic against both
the NotifyConfigChanged pointer swap and a second rotation. The loser of the
race observes the winner's refreshed created_at and does nothing.
Since s.mu is not reentrant, persistConfig splits into a locking wrapper and
persistConfigLocked, which callers already holding the lock use.
Test: eight concurrent rotations on one config must yield exactly one identity,
and the id left in memory (the one the heartbeat would report) must be the id on
disk. Verified to fail — as a detected data race — without the lock.
* fix(telemetry): close the remaining config data races on the heartbeat paths
The schema-v9 work made buildHeartbeat snapshot-safe, but two writers of the
same live *config.Config stayed outside s.mu. NotifyConfigChanged swaps that
pointer under the mutex, so an unlocked read or mutation of it is a genuine
data race, not a stale-but-safe view.
advanceUpgradeFunnel resolved the pointer under the lock and then mutated
cfg.Telemetry.LastReportedVersion after releasing it, while buildHeartbeat read
that same field unlocked. Both run concurrently in production: the heartbeat
loop advances the cursor right after a successful send, and BuildPayload is
exported and served from the REST handler behind `mcpproxy telemetry
show-payload`. Reproduced under -race, then fixed by moving the whole
resolve -> check -> mutate -> persist pass under s.mu (advanceUpgradeFunnelOnce)
and reading the four cfg.Telemetry scalars the payload reports in one locked
pass (telemetryCursor).
ensureAnonymousID had the same shape and one hazard more: it read s.config
unlocked and called config.SaveConfig on it directly, bypassing the liveness
check persistConfigLocked exists to enforce. Because Start() is launched with
`go` from runtime/lifecycle.go it overlaps the daemon's config-reload path, so
that write could put a stale whole-config file on top of a just-applied one. It
now runs as one locked pass via persistConfigLocked, retrying against the new
live config if the pointer moved, and Start's own telemetry-enabled check reads
through liveConfig.
Behaviour is unchanged on the single-threaded path; a genuine write error still
keeps the in-memory id so the process reports one stable identity.
Regression tests: TestAdvanceUpgradeFunnelConfigRace,
TestEnsureAnonymousIDConfigRace, TestEnsureAnonymousIDDoesNotSaveStaleConfig.
* fix(telemetry): resolve the telemetry-enabled check under the service mutex
preflightSink snapshotted s.config under s.mu but called
EffectiveTelemetryEnabled(cfg) after unlocking, and that path dereferences
cfg.Telemetry via config.IsTelemetryEnabled. Both ensureAnonymousIDOnce and
advanceUpgradeFunnelOnce install that same pointer under s.mu when the config
arrived without a telemetry block — the ordinary fresh-install shape. A locked
write paired with an unlocked read is still a data race, and the two sides here
are the request path (Record*, through preflightSink) and the heartbeat loop,
which run concurrently by construction.
Replaced the snapshot-then-dereference with telemetryEnabledLive(), which holds
s.mu across the whole evaluation. EffectiveTelemetryEnabled only reads env vars
and config fields, so it cannot re-enter the Service. Start()'s own
telemetry-enabled check had the same shape and now goes through the same helper.
Regression test: TestPreflightSinkTelemetryPointerRace.1 parent a52e90a commit 5e4698a
22 files changed
Lines changed: 1689 additions & 108 deletions
File tree
- docs/features
- internal
- httpapi
- runtime
- server
- telemetry
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
| 13 | + | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
| 42 | + | |
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
| |||
161 | 162 | | |
162 | 163 | | |
163 | 164 | | |
164 | | - | |
| 165 | + | |
165 | 166 | | |
166 | 167 | | |
167 | 168 | | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
168 | 185 | | |
169 | 186 | | |
170 | 187 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
127 | 127 | | |
128 | 128 | | |
129 | 129 | | |
130 | | - | |
131 | | - | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
132 | 133 | | |
133 | 134 | | |
134 | 135 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
195 | 196 | | |
196 | 197 | | |
197 | 198 | | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
198 | 203 | | |
199 | 204 | | |
200 | 205 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
869 | 869 | | |
870 | 870 | | |
871 | 871 | | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
872 | 876 | | |
873 | 877 | | |
874 | 878 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
105 | 111 | | |
106 | 112 | | |
107 | 113 | | |
| |||
232 | 238 | | |
233 | 239 | | |
234 | 240 | | |
235 | | - | |
236 | | - | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
237 | 247 | | |
238 | 248 | | |
239 | 249 | | |
| |||
302 | 312 | | |
303 | 313 | | |
304 | 314 | | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
305 | 357 | | |
306 | 358 | | |
307 | 359 | | |
| |||
483 | 535 | | |
484 | 536 | | |
485 | 537 | | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
486 | 541 | | |
487 | 542 | | |
488 | 543 | | |
| |||
559 | 614 | | |
560 | 615 | | |
561 | 616 | | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
562 | 623 | | |
563 | 624 | | |
564 | 625 | | |
| |||
0 commit comments