Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions docs/architecture/tla/ExceptionCapture.tla
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---- MODULE ExceptionCapture ----
\* Gate: TLA_JAR=<jar> ./check.sh ExceptionCaptureMC ExceptionCopyMC
\* Source: TelemetryService.captureException. Sink: each registered TelemetryClient.captureException.
\* Failures: repeats emit, distinct fields collide, blocked calls change state, or the cache grows.
\* Rules: WindowDecision, FieldsDistinct, BlockedUnchanged, and Bounded prevent those failures.
\* Expected before simplification: the shared cache refresh preserves all rules. No new code bug is expected.
\* Initial rules preceded review comments, but followed inspection of the original fix.
\* Review: PR 2330 discussion_r4486070563 is covered by WindowDecision, including window expiry.
\* Review counts for this module: 1 covered, 0 added, 0 gap. Copy findings are in ExceptionCopy.
\* Prior replay found delimiter collisions. The JSON tuple fix keeps (a|b, c) distinct from (a, b|c).
\* Prior replay removed identity, window, and size guards. Each real-code regression test failed.
\* Property tests: TelemetryService.exception-single-capture.test.ts checks 256 four-call traces and cache eviction.
\* A time advance saturates cache ages at expiry. It is separate from a capture call.
\* Instance identity is separate from field equality. Two instances can have the same fields.
\* The window applies to retained entries. Expiry or eviction permits a fresh instance to emit again.
\* ready is the service gate, not client opt-in. Client filtering and network success are outside the model.
\* Historical cost: about 6 minutes for the first spec, replay, tests, and fixes.
\* Result: ExceptionCaptureMC holds in 1,079 states. Joined, NoIdentity, NoWindow, and Unbounded each fail.
\* Cost: about 4 minutes across both new models, the code refactor, and validation. Each TLC run takes less than 2 seconds.
\* Null result: no new production bug. The shared refresh passes the property tests and eviction regression.
\* WindowDecision covers repeat identity, so the redundant identity history flag was removed.
\* BlockedUnchanged is an action property; it needs no history flag. The instance set bounds marked.
\* The copy model has its own states. Combining it with this cache would add unrelated state combinations.
\* Guidance: https://lamport.azurewebsites.net/tla/book-02-02-27.pdf (Chapter 7).
\* MC layout: https://github.com/tlaplus/vscode-tlaplus/blob/master/resources/knowledgebase/tlc-config-files.md
\* TLC checks the model, not code correctness. The prior trace-limited model visited 67 states.

EXTENDS Naturals, Sequences
CONSTANTS Instances, Fields, Fingerprint(_), Window, Limit, Gaps,
IdentityGuard, WindowGuard, SizeGuard
VARIABLES marked, cache, last
vars == <<marked, cache, last>>

NoCall == [kind |-> "tick", ready |-> FALSE, marked |-> FALSE, recent |-> FALSE, emitted |-> FALSE]
Drop(c, k) == SelectSeq(c, LAMBDA entry : entry.key # k)
Recent(c, k) == \E n \in 1..Len(c) : c[n].key = k /\ c[n].age < Window
Refresh(c, k) == Append(Drop(c, k), [key |-> k, age |-> 0])
Trim(c) == IF SizeGuard /\ Len(c) > Limit THEN Tail(c) ELSE c

Init == marked = {} /\ cache = <<>> /\ last = NoCall

Call(i, ready) ==
LET key == Fingerprint(Fields[i])
known == i \in marked
recent == Recent(cache, key)
accepts == ready /\ (~IdentityGuard \/ ~known)
IN /\ marked' = IF ready THEN marked \cup {i} ELSE marked
/\ cache' = IF accepts THEN Trim(Refresh(cache, key)) ELSE cache
/\ last' = [kind |-> "call", ready |-> ready, marked |-> known, recent |-> recent,
emitted |-> accepts /\ (~WindowGuard \/ ~recent)]

Tick(gap) ==
/\ cache' = [n \in 1..Len(cache) |->
[cache[n] EXCEPT !.age = IF @ + gap >= Window THEN Window ELSE @ + gap]]
/\ last' = NoCall
/\ UNCHANGED marked

Next == (\E i \in Instances, ready \in BOOLEAN : Call(i, ready)) \/ (\E gap \in Gaps : Tick(gap))
Spec == Init /\ [][Next]_vars

TypeOK == /\ marked \subseteq Instances
/\ cache \in Seq([key : {Fingerprint(Fields[i]) : i \in Instances}, age : 0..Window])
/\ last \in [kind : {"call", "tick"}, ready : BOOLEAN, marked : BOOLEAN,
recent : BOOLEAN, emitted : BOOLEAN]
WindowDecision == last.kind = "call" =>
(last.emitted = (last.ready /\ ~last.marked /\ ~last.recent))
FieldsDistinct == \A i \in marked, j \in Instances :
Fields[i] # Fields[j] => Fingerprint(Fields[i]) # Fingerprint(Fields[j])
Bounded == Len(cache) <= Limit
\* A blocked call leaves both policy stores unchanged. Time advances are separate actions.
BlockedUnchanged == [][(last'.kind = "call" /\ ~last'.ready) => UNCHANGED <<marked, cache>>]_vars
====
14 changes: 14 additions & 0 deletions docs/architecture/tla/ExceptionCaptureMC.Joined.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
SPECIFICATION Spec
CONSTANTS
Instances <- MCInstances
Fields <- MCFields
Fingerprint <- JoinedKey
Window <- MCWindow
Limit <- MCLimit
Gaps <- MCGaps
IdentityGuard <- On
WindowGuard <- On
SizeGuard <- On
INVARIANTS TypeOK WindowDecision FieldsDistinct Bounded
PROPERTY BlockedUnchanged
CHECK_DEADLOCK FALSE
14 changes: 14 additions & 0 deletions docs/architecture/tla/ExceptionCaptureMC.NoIdentity.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
SPECIFICATION Spec
CONSTANTS
Instances <- MCInstances
Fields <- MCFields
Fingerprint <- TupleKey
Window <- MCWindow
Limit <- MCLimit
Gaps <- MCGaps
IdentityGuard <- Off
WindowGuard <- On
SizeGuard <- On
INVARIANTS TypeOK WindowDecision FieldsDistinct Bounded
PROPERTY BlockedUnchanged
CHECK_DEADLOCK FALSE
14 changes: 14 additions & 0 deletions docs/architecture/tla/ExceptionCaptureMC.NoWindow.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
SPECIFICATION Spec
CONSTANTS
Instances <- MCInstances
Fields <- MCFields
Fingerprint <- TupleKey
Window <- MCWindow
Limit <- MCLimit
Gaps <- MCGaps
IdentityGuard <- On
WindowGuard <- Off
SizeGuard <- On
INVARIANTS TypeOK WindowDecision FieldsDistinct Bounded
PROPERTY BlockedUnchanged
CHECK_DEADLOCK FALSE
14 changes: 14 additions & 0 deletions docs/architecture/tla/ExceptionCaptureMC.Unbounded.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
SPECIFICATION Spec
CONSTANTS
Instances <- MCInstances
Fields <- MCFields
Fingerprint <- TupleKey
Window <- MCWindow
Limit <- MCLimit
Gaps <- MCGaps
IdentityGuard <- On
WindowGuard <- On
SizeGuard <- Off
INVARIANTS TypeOK WindowDecision FieldsDistinct Bounded
PROPERTY BlockedUnchanged
CHECK_DEADLOCK FALSE
14 changes: 14 additions & 0 deletions docs/architecture/tla/ExceptionCaptureMC.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
SPECIFICATION Spec
CONSTANTS
Instances <- MCInstances
Fields <- MCFields
Fingerprint <- TupleKey
Window <- MCWindow
Limit <- MCLimit
Gaps <- MCGaps
IdentityGuard <- On
WindowGuard <- On
SizeGuard <- On
INVARIANTS TypeOK WindowDecision FieldsDistinct Bounded
PROPERTY BlockedUnchanged
CHECK_DEADLOCK FALSE
16 changes: 16 additions & 0 deletions docs/architecture/tla/ExceptionCaptureMC.tla
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---- MODULE ExceptionCaptureMC ----
\* Finite inputs and guard substitutions for ExceptionCapture. No trace counter is needed.
\* Two instances share fields. Two different tuples collide under delimiter joining. A third key forces eviction.
EXTENDS ExceptionCapture

MCInstances == 1..4
MCFields == [i \in MCInstances |->
CASE i \in {1, 2} -> <<"a|b", "c">> [] i = 3 -> <<"a", "b|c">> [] OTHER -> <<"a", "c">>]
TupleKey(fields) == fields
JoinedKey(fields) == IF fields \in {MCFields[1], MCFields[3]} THEN MCFields[1] ELSE fields
MCWindow == 2
MCLimit == 2
MCGaps == {1, 2, 3}
On == TRUE
Off == FALSE
====
47 changes: 47 additions & 0 deletions docs/architecture/tla/ExceptionCopy.tla
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---- MODULE ExceptionCopy ----
\* Source: PostHogTelemetryClient.captureException(error). Sink: PostHog.captureException(payload).
\* Failures: normalization changes the caller error, loses error type or children, or sends the wrong message.
\* Rules: CallerMessage, SourceShape, and NormalizedMessage compare source and delivered records directly.
\* The transform is an oracle. ExceptionCopyMC supplies fixed and broken transforms for the finite check.
\* Expected before simplification: fixed records pass; mutation and prototype loss each fail.
\* Review: PR 2330 discussion_r4486068932 added CallerMessage to the original combined model.
\* Review: PR 2330 discussion_r4638627894 added SourceShape after the copy lost built-in prototypes.
\* Review counts for this module: 0 covered, 2 added, 0 gap. Dedupe feedback is in ExceptionCapture.
\* Replay: the real client changed three caller messages before the detached-copy fix.
\* Replay: TypeError, RangeError, and AggregateError each lost their prototype for three test messages.
\* Property tests: PostHogTelemetryClient.message-preservation.test.ts checks messages, types, stacks, causes, and children.
\* proto represents the inherited error type. Only an AggregateError exposes children to the sink.
\* Fields stand for data or references. This model does not copy JavaScript descriptors or PostHog parsing.
\* Historical cost: about 2 minutes for the prototype-loss spec, replay, fix, and checks.
\* Result: ExceptionCopyMC holds in 32 states. Mutating fails CallerMessage; PlainCopy fails SourceShape.
\* Cost: included in ExceptionCapture. Each TLC run takes less than 2 seconds. No new production bug was found.
\* Removed caller and shape success flags. The invariants now inspect the actual transform output.
\* Guidance: https://lamport.azurewebsites.net/tla/book-02-02-27.pdf (Chapter 7).
\* TLC checks the model, not code correctness. Real-code tests provide the source-to-sink replay.

EXTENDS Naturals, Sequences
CONSTANTS Inputs, Messages, Transform(_, _)
VARIABLES source, raw, caller, delivered
vars == <<source, raw, caller, delivered>>

Children(error) == IF error.proto = "AggregateError" THEN error.own.children ELSE <<>>
Init == /\ source \in Inputs /\ raw \in Messages
/\ caller = source /\ delivered = <<>>
Capture ==
LET output == Transform(source, raw)
IN /\ Len(delivered) = 0
/\ caller' = output.caller /\ delivered' = <<output.payload>>
/\ UNCHANGED <<source, raw>>
Spec == Init /\ [][Capture]_vars

TypeOK == /\ source \in Inputs /\ raw \in Messages /\ caller \in Inputs
/\ delivered \in Seq([proto : {error.proto : error \in Inputs}, own : {error.own : error \in Inputs}])
/\ Len(delivered) <= 1
CallerMessage == caller = source
SourceShape == Len(delivered) = 1 =>
/\ delivered[1].proto = source.proto
/\ delivered[1].own.stack = source.own.stack
/\ delivered[1].own.cause = source.own.cause
/\ Children(delivered[1]) = Children(source)
NormalizedMessage == Len(delivered) = 1 => delivered[1].own.message = raw
====
7 changes: 7 additions & 0 deletions docs/architecture/tla/ExceptionCopyMC.Mutating.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
SPECIFICATION Spec
CONSTANTS
Inputs <- MCInputs
Messages <- MCMessages
Transform <- Mutating
INVARIANTS TypeOK CallerMessage SourceShape NormalizedMessage
CHECK_DEADLOCK FALSE
7 changes: 7 additions & 0 deletions docs/architecture/tla/ExceptionCopyMC.PlainCopy.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
SPECIFICATION Spec
CONSTANTS
Inputs <- MCInputs
Messages <- MCMessages
Transform <- PlainCopy
INVARIANTS TypeOK CallerMessage SourceShape NormalizedMessage
CHECK_DEADLOCK FALSE
7 changes: 7 additions & 0 deletions docs/architecture/tla/ExceptionCopyMC.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
SPECIFICATION Spec
CONSTANTS
Inputs <- MCInputs
Messages <- MCMessages
Transform <- Detached
INVARIANTS TypeOK CallerMessage SourceShape NormalizedMessage
CHECK_DEADLOCK FALSE
15 changes: 15 additions & 0 deletions docs/architecture/tla/ExceptionCopyMC.tla
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---- MODULE ExceptionCopyMC ----
\* Finite source records and copy transforms for ExceptionCopy. The two broken transforms are sabotage checks.
EXTENDS ExceptionCopy

MCNames == {"Error", "TypeError", "RangeError", "AggregateError"}
MCMessages == {"wrapped", "raw"}
MCInputs == { [proto |-> kind,
own |-> [message |-> message, stack |-> "source stack", cause |-> "cause reference",
children |-> IF kind = "AggregateError" THEN <<"child reference">> ELSE <<>>]] :
kind \in MCNames, message \in MCMessages }
Normalized(error, message) == [error EXCEPT !.own.message = message]
Detached(error, message) == [caller |-> error, payload |-> Normalized(error, message)]
Mutating(error, message) == [caller |-> Normalized(error, message), payload |-> Normalized(error, message)]
PlainCopy(error, message) == [caller |-> error, payload |-> [Normalized(error, message) EXCEPT !.proto = "Error"]]
====
11 changes: 8 additions & 3 deletions packages/telemetry/src/PostHogTelemetryClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,13 @@ export class PostHogTelemetryClient extends BaseTelemetryClient {
mergedProperties = { ...extractedProperties, ...additionalProperties }
}

// Override the error message with the extracted error message.
error.message = errorMessage
// Normalize a detached error so telemetry cannot change the caller-facing message.
const stack = error.stack
const telemetryError = new Error()
Object.setPrototypeOf(telemetryError, Object.getPrototypeOf(error))
Object.defineProperties(telemetryError, Object.getOwnPropertyDescriptors(error))
telemetryError.stack = stack
telemetryError.message = errorMessage

const provider = this.providerRef?.deref()
let telemetryProperties: TelemetryProperties | undefined = undefined
Expand All @@ -135,7 +140,7 @@ export class PostHogTelemetryClient extends BaseTelemetryClient {
$app_version: telemetryProperties?.appVersion,
}

this.client.captureException(error, this.distinctId, exceptionProperties)
this.client.captureException(telemetryError, this.distinctId, exceptionProperties)
}

/**
Expand Down
92 changes: 92 additions & 0 deletions packages/telemetry/src/TelemetryService.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import {
TelemetryEventName,
type TelemetrySetting,
type ToolUsage,
isApiProviderError,
} from "@roo-code/types"

/**
Expand All @@ -27,6 +28,45 @@ const CIRCUIT_BREAKER_WINDOW_MS = 10 * 60 * 1000
/** How long a tripped breaker stays tripped before allowing captures again. */
const CIRCUIT_BREAKER_COOLDOWN_MS = 10 * 60 * 1000

/**
* How long an exception fingerprint is remembered for the single-capture
* policy below. Provider retry loops re-enter the provider per attempt and each
* attempt builds a fresh error, so identical errors re-thrown across attempts
* collapse into the first capture within this window. Each fresh identical
* error refreshes the window. A gap of 30 seconds or more permits a new event.
* Exponential backoff, server retry delays, and request duration can exceed
* this window, so a persistent retry chain can emit again after such a gap.
* This bounds bursts; it does not identify an entire retry chain.
*/
const EXCEPTION_DEDUPE_WINDOW_MS = 30 * 1000
Comment thread
zoomote[bot] marked this conversation as resolved.
Resolved

/**
* Upper bound on remembered exception fingerprints (oldest evicted) so a long
* session streaming many distinct errors cannot grow the map without bound.
*/
const EXCEPTION_DEDUPE_CACHE_LIMIT = 100

/**
* Stable identity for an exception under the single-capture policy: everything
* providers put on an ApiProviderError (provider, model, operation, code,
* message), or type plus message for other errors. Two errors with the same
* fingerprint within the dedupe window are treated as one occurrence.
*/
function exceptionFingerprint(error: Error): string {
if (isApiProviderError(error)) {
return JSON.stringify([
"ApiProviderError",
error.provider,
error.modelId,
error.operation,
error.errorCode ?? "",
error.message,
])
}

return JSON.stringify([error.name, error.message])
}

/**
* Upper bound applied separately to each phase of shutdown(): draining in-flight capture
* calls, then awaiting client.shutdown(). deactivate() awaits shutdown() before terminal
Expand All @@ -49,6 +89,13 @@ export class TelemetryService {
private guardedEventOccurrences = new Map<TelemetryEventName, number[]>()
private trippedUntil = new Map<TelemetryEventName, number>()

// Exception fingerprint -> time of the latest capture attempt, oldest first.
// Backs the single-capture policy in captureException(); entries are re-dated
// (and moved to the back) on every dedupe hit so the window slides while an
// identical error keeps recurring, and the oldest entry is the eviction
// candidate when the cache outgrows EXCEPTION_DEDUPE_CACHE_LIMIT.
private recentExceptionFingerprints = new Map<string, number>()

// In-flight client.capture()/captureException() promises. captureEvent/captureException are
// synchronous (void-returning) for callers, but the underlying client calls are async (e.g.
// PostHogTelemetryClient awaits property enrichment before enqueueing). Tracked here so
Expand Down Expand Up @@ -171,6 +218,51 @@ export class TelemetryService {
return
}

// Single-capture policy (issue #513): one error occurrence must produce one
// $exception event, but provider catch blocks capture and re-throw, and the
// retry loop in Task#attemptApiRequest re-enters the provider per attempt,
// so the same failure reaches this method several times. Two guards, in
// order:
//
// 1. Identity. An ApiProviderError is marked on its first capture; any
// later capture of the same instance (enclosing catch, re-throw) is a
// no-op. Marking happens only after the readiness check above, so an
// error captured while telemetry is down can still be captured later.
//
// 2. Fingerprint window. Each retry attempt constructs a fresh
// ApiProviderError, which identity cannot see, so identical errors
// within EXCEPTION_DEDUPE_WINDOW_MS collapse into their first capture.
if (isApiProviderError(error)) {
if (error.captured) {
return
}

error.captured = true
}

const fingerprint = exceptionFingerprint(error)
const now = Date.now()
const lastSeenAt = this.recentExceptionFingerprints.get(fingerprint)

// Every fresh instance refreshes recency, including a duplicate inside the window.
// Delete before set so Map iteration keeps the least recently observed key first.
this.recentExceptionFingerprints.delete(fingerprint)
this.recentExceptionFingerprints.set(fingerprint, now)

while (this.recentExceptionFingerprints.size > EXCEPTION_DEDUPE_CACHE_LIMIT) {
const oldest = this.recentExceptionFingerprints.keys().next().value

if (oldest === undefined) {
break
}

this.recentExceptionFingerprints.delete(oldest)
}

if (lastSeenAt !== undefined && now - lastSeenAt < EXCEPTION_DEDUPE_WINDOW_MS) {
return
}

this.clients.forEach((client) =>
this.trackPendingClientCall(client.captureException(error, additionalProperties)),
)
Expand Down
Loading
Loading