Skip to content

feat(telemetry): heartbeat v9 — TPA gate/prompt scan counters + trust_mode distribution - #1029

Merged
github-actions[bot] merged 9 commits into
mainfrom
feat/telemetry-v9-tpa-funnel
Aug 24, 2026
Merged

github-actions[bot] merged 9 commits into
mainfrom
feat/telemetry-v9-tpa-funnel

Conversation

@Dumbris

@Dumbris Dumbris commented Aug 23, 2026

Copy link
Copy Markdown
Member

What

Heartbeat schema v9, making the TPA funnel measurable.

  • Two new delta counters in the tpa_scanner sub-object — tool_change_gate_scans and prompt_scans — recorded through the telemetry counter registry (same shape as RecordTPAScanCompleted: window scoped, reset only after an accepted 2xx send).
  • New state field trust_mode_distribution: {auto, scan, manual} counts of configured servers by ServerConfig.EffectiveTrustMode(), recomputed from the live config at heartbeat build time.
  • schema_version bumped 8 → 9.

Why

The v8 counters only see scan jobs, which most installs never start. The two TPA detection paths that actually run for ordinary users emitted nothing at all, so the fleet read as "the scanner never runs":

  • internal/runtime.scanChangeIsClean — the synchronous trust_mode: scan tool-change gate
  • internal/server.scanAggregatedPrompts — the prompt-poisoning filter

Both called scanner.ScanToolMetadataVerdict directly with no counter. And there was no fleet-wide view of which trust tier installs sit in — which is also the denominator the gate counter needs, since only scan-mode servers can produce a gate scan.

Key decisions

  • Counts of invocations, not outcomes. Neither counter accepts the server, the tool/prompt, the verdict, or the matched checks. prompt_scans is per prompt actually scanned (a malformed, unqualified prompt name short-circuits before the scanner and is not counted).
  • Omission posture preserved. tpa_scanner is still dropped entirely when every counter — v8 and v9 — is zero, so an install that scans nothing stays shape-identical to a v7 payload. trust_mode_distribution is a state field and is always emitted (all three keys, zeros included), matching the server_protocol_counts convention.
  • Do not sum the two counter groups: units differ (one changed tool / one prompt vs. one scan job). Documented on the struct and in docs/features/telemetry.md.
  • Anonymity backstop widened: tpa_scanner's key whitelist grows by the two counters (rule v8_field_invalid), and the histogram gets its own rule trust_mode_field_invalid — fixed auto|scan|manual keys with non-negative integer counts, or the heartbeat is blocked instead of sent. Trust tiers are resolved through EffectiveTrustMode(), so an empty/typo'd mode and the legacy auto_approve_tool_changes / skip_quarantine fields all fold into the enum before counting.

No frontend contract or OAS surface touches this payload, so no contracts.ts regen or make swagger was needed.

Follow-ups (not in this PR)

  • Shutdown flush of the counters — a window's worth of gate/prompt scans is still lost on exit, same as every other registry counter.
  • Dashboard-side (mcpproxy-dash) ingestion of tool_change_gate_scans, prompt_scans, and trust_mode_distribution.

…_mode distribution

The v8 tpa_scanner counters only see scan JOBS, which most installs never
start. The two TPA detection paths that run synchronously for ordinary users
emitted nothing at all, so the fleet read as "the scanner never runs":

- the trust_mode:scan tool-change gate (runtime.scanChangeIsClean)
- the aggregated-prompt poisoning filter (server.scanAggregatedPrompts)

Both now increment a delta counter in the tpa_scanner sub-object
(tool_change_gate_scans / prompt_scans) via the telemetry counter registry,
with the same window and reset-after-accepted-send semantics as every other
counter. Counts of invocations only — never the server, the tool/prompt, or
the verdict. The sub-object is still omitted when all counters (v8 and v9)
are zero.

Adds trust_mode_distribution: a state field counting configured servers per
EffectiveTrustMode(), keyed by the fixed auto|scan|manual enum and recomputed
from the live config on each heartbeat. It is the denominator the gate counter
needs (only "scan" servers can produce a gate scan) and the first fleet-wide
view of trust-tier adoption.

schema_version bumped to 9. The anonymity scanner widens the tpa_scanner key
whitelist and gains rule trust_mode_field_invalid so a producer-side
regression leaking a server name as a histogram key blocks the send.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 23, 2026 •

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 03d4154
Status: ✅  Deploy successful!
Preview URL: https://c688454a.mcpproxy-docs.pages.dev
Branch Preview URL: https://feat-telemetry-v9-tpa-funnel.mcpproxy-docs.pages.dev

View logs

@codecov-commenter

codecov-commenter commented Aug 23, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 86.88525% with 24 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/telemetry/telemetry.go 80.70% 14 Missing and 8 partials ⚠️
internal/telemetry/trust_mode.go 89.47% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Aug 23, 2026 •

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: feat/telemetry-v9-tpa-funnel

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (26 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (15 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (25 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (23 MB)
  • installer-dmg-darwin-arm64 (20 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 32687942431 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

Cross-model review of the v9 payload flagged the trust_mode_distribution
builder reading s.config unsynchronized. NotifyConfigChanged replaces that
pointer wholesale under s.mu on every live config reload (REST apply, disk
watcher), so the read is a genuine data race — and it was not new: eight
sibling reads in buildHeartbeat, plus maybeRotateAnonymousID and
persistConfig, had the same exposure. trust_mode_distribution just made it
visible by walking cfg.Servers.

buildHeartbeat now resolves the live config exactly once through a new
liveConfig() accessor (lock, read, release — the mutex is non-reentrant and
preflightSink takes it, so it must not be held across the build) and every
downstream read uses that snapshot. maybeRotateAnonymousID and persistConfig
take the config as a parameter so the rotation mutates and persists the same
pointer the payload reported, instead of whatever s.config points at by then.

Adds a regression test that hammers NotifyConfigChanged against concurrent
buildHeartbeat calls; under -race it fails on the pre-fix code and also
asserts the histogram is never torn across a swap (all three fixed keys, two
servers total).
…accessor

advanceUpgradeFunnel runs on the heartbeat loop right after a 2xx send and
mutates config.Telemetry.LastReportedVersion, but it still read s.config
directly -- the same race with the NotifyConfigChanged pointer swap that
buildHeartbeat was just fixed for. Route it through liveConfig() so the whole
heartbeat build-and-send path reads the live config through one accessor.
… out

Round 2 of the cross-model review caught a regression in round 1's fix. The
heartbeat now works from a config SNAPSHOT, and persistConfig writes the WHOLE
config file — so an in-flight heartbeat could save its stale snapshot over a
config the user had just applied: snapshot A, user applies B (already on disk),
the in-flight anonymous-ID rotation saves A, B is silently rolled back.

persistConfig now writes only while the config it was handed is still s.config,
with the liveness check and the write under one s.mu hold so the swap cannot
slip between them. Skipping is safe and self-healing: every caller's mutation is
idempotent, so the next heartbeat re-evaluates it against the live config and
persists then. advanceUpgradeFunnel drops its open-coded SaveConfig and goes
through the same guarded path.

Two regression tests, both verified to fail with the guard disabled: a direct
persistConfig-after-swap case (asserting the guard is a liveness check, not a
blanket refusal — the live config still writes) and the end-to-end rotation
case that reaches persistConfig through maybeRotateAnonymousID.
Round 3 of the cross-model review found the round-2 guard was correct but that
"skipping the write is safe because the mutation is idempotent" was not true for
two of its callers. persistConfig now reports whether it wrote, and both
callers act on that:

- Anonymous-ID rotation is all-or-nothing. The caller transmits the snapshot's
  anonymous_id in the payload it is building, so an id that was never written
  to disk must never leave the machine: one annual rotation would otherwise
  surface as TWO identities (this heartbeat's unpersisted id, then the id the
  next heartbeat rotates the live config to) and fragment the install's
  telemetry continuity. On a skipped write the snapshot is restored.

- The upgrade-funnel cursor is not self-healing. Leaving last_reported_version
  unadvanced makes the next heartbeat report the same previous_version again,
  double-counting one upgrade, so the advance is redone against the new live
  config when the guarded write was skipped.

persistConfig also now distinguishes "nothing to persist" from "write skipped":
a service with no cfgPath (in-memory/CLI use) has no on-disk state for the
mutation to be inconsistent with, so it reports success and callers keep their
mutation. Without that split the rotation rolled itself back in every test and
CLI service — caught by TestIDRotatesAfter365Days.

The residual clobber window the review also raised is documented on
persistConfig rather than papered over: the daemon's config writers save the
file BEFORE calling NotifyConfigChanged, so in that gap the old pointer still
looks live. Closing it needs single-writer ownership of the config file, a
config-layer change out of scope here; the guard narrows the exposure, it does
not eliminate it.

Tests: rotation rollback and the funnel end-state after a swap, both verified
to fail against the unfixed code.
Round 4 of the cross-model review found the rotation rollback was itself not
concurrency-safe. BuildPayload is exported and served from an HTTP handler
(internal/httpapi), so a request can build a heartbeat concurrently with the
heartbeat loop and two rotations can race on the same config. Interleaved, one
captures the OTHER's freshly generated id as its "previous" value and restores
that on rollback — leaving a never-persisted id in the config the payload then
transmits, which is exactly the identity fragmentation the rollback was added
to prevent.

maybeRotateAnonymousID now holds s.mu across its whole check → generate →
mutate → persist-or-rollback sequence, so the rotation is atomic against both
the NotifyConfigChanged pointer swap and a second rotation. The loser of the
race observes the winner's refreshed created_at and does nothing.

Since s.mu is not reentrant, persistConfig splits into a locking wrapper and
persistConfigLocked, which callers already holding the lock use.

Test: eight concurrent rotations on one config must yield exactly one identity,
and the id left in memory (the one the heartbeat would report) must be the id on
disk. Verified to fail — as a detected data race — without the lock.
…t paths

The schema-v9 work made buildHeartbeat snapshot-safe, but two writers of the
same live *config.Config stayed outside s.mu. NotifyConfigChanged swaps that
pointer under the mutex, so an unlocked read or mutation of it is a genuine
data race, not a stale-but-safe view.

advanceUpgradeFunnel resolved the pointer under the lock and then mutated
cfg.Telemetry.LastReportedVersion after releasing it, while buildHeartbeat read
that same field unlocked. Both run concurrently in production: the heartbeat
loop advances the cursor right after a successful send, and BuildPayload is
exported and served from the REST handler behind `mcpproxy telemetry
show-payload`. Reproduced under -race, then fixed by moving the whole
resolve -> check -> mutate -> persist pass under s.mu (advanceUpgradeFunnelOnce)
and reading the four cfg.Telemetry scalars the payload reports in one locked
pass (telemetryCursor).

ensureAnonymousID had the same shape and one hazard more: it read s.config
unlocked and called config.SaveConfig on it directly, bypassing the liveness
check persistConfigLocked exists to enforce. Because Start() is launched with
`go` from runtime/lifecycle.go it overlaps the daemon's config-reload path, so
that write could put a stale whole-config file on top of a just-applied one. It
now runs as one locked pass via persistConfigLocked, retrying against the new
live config if the pointer moved, and Start's own telemetry-enabled check reads
through liveConfig.

Behaviour is unchanged on the single-threaded path; a genuine write error still
keeps the in-memory id so the process reports one stable identity.

Regression tests: TestAdvanceUpgradeFunnelConfigRace,
TestEnsureAnonymousIDConfigRace, TestEnsureAnonymousIDDoesNotSaveStaleConfig.
… mutex

preflightSink snapshotted s.config under s.mu but called
EffectiveTelemetryEnabled(cfg) after unlocking, and that path dereferences
cfg.Telemetry via config.IsTelemetryEnabled. Both ensureAnonymousIDOnce and
advanceUpgradeFunnelOnce install that same pointer under s.mu when the config
arrived without a telemetry block — the ordinary fresh-install shape. A locked
write paired with an unlocked read is still a data race, and the two sides here
are the request path (Record*, through preflightSink) and the heartbeat loop,
which run concurrently by construction.

Replaced the snapshot-then-dereference with telemetryEnabledLive(), which holds
s.mu across the whole evaluation. EffectiveTelemetryEnabled only reads env vars
and config fields, so it cannot re-enter the Service. Start()'s own
telemetry-enabled check had the same shape and now goes through the same helper.

Regression test: TestPreflightSinkTelemetryPointerRace.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved (Model B): Paperclip review verdicts = ACCEPT and qa-gate green at this head SHA. Arming auto-merge; GitHub merges when all required checks pass.

@github-actions
github-actions Bot enabled auto-merge (squash) August 24, 2026 10:31
@github-actions
github-actions Bot merged commit 5e4698a into main Aug 24, 2026
40 checks passed
github-actions Bot pushed a commit that referenced this pull request Aug 25, 2026
* fix(telemetry): flush a final heartbeat on graceful shutdown

Telemetry counters are in-memory and reset only after an accepted (2xx)
send, so everything recorded after the 5-minute first heartbeat was lost
unless the process survived to the 24h tick — which most desktop sessions
do not. Deferred follow-up from #1029.

Stop() now performs ONE final heartbeat after joining the loop:

- It reuses sendHeartbeat's send+reset path verbatim, so the counter
  contract is unchanged — a failed/timed-out flush drops nothing, an
  accepted flush cannot double-count with the next start.
- It runs AFTER the loop is joined rather than from a second goroutine,
  so there is provably no concurrent sender and the flush cannot
  interleave with a tick.
- It uses a fresh context hard-bounded to 4s (Runtime.Close cancels the
  app context before Stop, and a dead endpoint must never hang a quit).
- It is skipped when telemetry is disabled by env/config, the build is
  non-semver, the user opted out, or nothing was recorded since the last
  accepted send (new CounterRegistry.HasPendingCounters).
- Anonymous-id rotation is suppressed on this path: rotation rewrites the
  whole config file, and shutdown is the wrong moment for that.

Also closes a pre-existing data race the flush widens: s.endpoint is
rewritten under s.mu by NotifyConfigChanged but was read unlocked by the
heartbeat, feedback, and opt-out-beacon paths. All four now read it
through liveEndpoint(), matching liveConfig().

* fix(telemetry): don't let the shutdown flush spend one-shot state

Cross-model review round 1 (opencode / gpt-5.6-sol) found three real defects
in the graceful-shutdown flush.

1. A failed flush destroyed the installer attribution. resolveLaunchSource
   clears installer_heartbeat_pending at BUILD time, before the send is known
   to have landed — a deliberate trade for the long-running loop, but it
   inverts on the flush: the process is exiting, so an offline or timed-out
   flush spent a one-shot that previously survived into the next run. The
   flush now leaves the flag armed and reports the runtime-detected source;
   the next start still claims "installer" exactly once. Generalized the
   rotate flag into consumeOneShots, which now gates both the 365-day
   anonymous-id rotation and the installer flag (send/buildHeartbeat*
   renamed accordingly).

2. TestFlushOnStopNoDoubleSendWithPeriodicTick was flaky — reproduced at
   -count=300 (got 4, want 2). Its premise was wrong: a tick aborted after
   the endpoint accepted it leaves counters pending, so the flush re-sends
   them. Delivery is at-least-once (undecidable client-side), which is the
   deliberate trade against dropping the window entirely. Split into two
   deterministic tests: the flush skips after an ACCEPTED tick, and joins +
   re-sends an unconfirmed one. Documented the semantics in code and docs.

3. TestFlushOnStopSkippedWhenOptedOut raced the opt-out beacon, which POSTs
   to the same /heartbeat ingest path (its comment claimed /opt-out). The
   recorder now classifies beacons by their `event` field and the test waits
   for the beacon before asserting no usage heartbeat followed.

Rejected as pre-existing/undecidable, with rebuttals: heartbeat-URL
validateTelemetryURL pinning (pre-existing sink, local config value),
buildHeartbeat not bounded by flushTimeout (the joined tick already had the
same exposure), and the opt-out latch race (the flush's telemetryEnabledLive
plus the pre-transmit optedOut re-check already close the described window).

* fix(telemetry): read the anonymous id under s.mu in the opt-out beacon

Cross-model review round 2 (opencode / gpt-5.6-sol). One genuine finding: the
second half of this PR's own s.endpoint drive-by was left undone.

EmitOptOutBeacon and SendOptOutBeacon both gate on s.config.GetAnonymousID()
read WITHOUT s.mu, two lines above the endpoint read that now takes it.
NotifyConfigChanged swaps s.config under s.mu, and GetAnonymousID dereferences
cfg.Telemetry — a pointer ensureAnonymousIDOnce / advanceUpgradeFunnelOnce also
install under s.mu — so snapshotting the pointer would not have been enough
either; the whole read has to be inside the critical section, exactly as
telemetryEnabledLive documents. NotifyConfigChanged itself launches the beacon
goroutine, so the writer and the reader are one call apart.

Added liveAnonymousID() alongside liveConfig()/liveEndpoint() and routed both
gates through it. TestOptOutBeaconReadsAnonymousIDUnderLock reproduces the
race first (the detector flags both sites against optout.go:215) and passes
after. No unlocked s.config read remains outside a locked region.

Also re-raised and rejected again: the opt-out TOCTOU between the pre-transmit
optedOut check and client.Do. It is pre-existing on the periodic sender and
unchanged in character here, and the proposed fix — an emission mutex held
across the HTTP send — would make NotifyConfigChanged block the config-save
path on up to 10s of network I/O, breaking the "never blocks the caller"
invariant that goroutine exists to uphold.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants