Skip to content

[Fix] PostHog captures one exception per API error - #2330

Draft
zoomote[bot] wants to merge 3 commits into
mainfrom
fix/telemetry-single-capture-513
Draft

zoomote[bot] wants to merge 3 commits into
mainfrom
fix/telemetry-single-capture-513

Conversation

@zoomote

@zoomote zoomote Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

​Created by Roomote. Follow up by mentioning @roomote, in the web UI, or in Discord.

Related GitHub Issue

Closes: #513

Description

Providers caught API errors, sent one $exception event, and re-threw the same failure. The retry loop in Task#attemptApiRequest re-entered the provider on every attempt. One persistent API error produced three to six $exception events. PR #1702 fixed Bugs 1 and 3 of issue #513. This PR fixes Bug 2 and Bug 4 and adds client-side deduplication.

This PR picks Option B from the issue. TelemetryService#captureException now enforces one capture per error occurrence with two guards:

  1. Identity. ApiProviderError gains a captured flag. The first capture sets it. Re-throws of the same instance are no-ops. This guard also covers Bug 4.
  2. Fingerprint window. Each retry attempt builds a fresh error object, so identity cannot see it. Identical errors (provider, model, operation, code, message) inside a 30-second sliding window collapse into the first capture. The cache holds at most 100 fingerprints, oldest evicted.

Providers keep their catch blocks and their captures. No telemetry is removed. The issue asked for an audit of capture paths outside the retry loop before this choice. completePrompt and similar paths capture once at the provider and keep doing so. The policy does not drop that telemetry.

Bug 4: OpenRouterHandler#handleStreamingError stays the single capture point for mid-stream OpenRouter errors. It now throws the same ApiProviderError instance it captured, so an enclosing capture of that instance is a no-op through the flag. The caller-facing message text is unchanged. One payload note: the PostHog message now carries the OpenRouter API Error <code>: prefix, because one object is both captured and thrown.

Test Procedure

New tests:

  • packages/telemetry/src/__tests__/TelemetryService.exception-single-capture.test.ts. Pins the flag, the window, window expiry, sliding refresh, distinct errors, plain errors, and the not-ready path. Eight tests.
  • src/core/task/__tests__/Task.spec.ts adds a fault-injection test. It drives the real retry loop in attemptApiRequest. The provider seam captures and throws on attempts one and two and recovers on attempt three. The test asserts exactly one client captureException call and a successful recovery.
  • The same file adds a mistake-limit guard test. It asserts the named Consecutive Mistake Error event fires and captureException never runs. ConsecutiveMistakeError therefore never reaches $exception.
  • src/api/providers/__tests__/openrouter.spec.ts expects the captured message with the OpenRouter API Error <code>: prefix on the four handleStreamingError paths.

Commands and results:

  • pnpm --filter @roo-code/telemetry test: 59 tests pass.
  • pnpm --dir src exec vitest run api/providers/__tests__/openrouter.spec.ts: 41 tests pass.
  • pnpm --dir src exec vitest run core/task/__tests__/Task.spec.ts: 204 tests pass.
  • pnpm test from the repository root: 565 test files pass, zero failures. Two unrelated extension.spec.ts tests flaked once under full-suite parallelism. The same flake reproduces on a clean main tree and is not caused by this change.
  • tsc --noEmit passes in src, @roo-code/types, and @roo-code/telemetry.
  • ESLint with --prune-suppressions passes on every changed file. No suppression count increases.

Pre-Submission Checklist

  • Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
  • Scope: My changes are focused on the linked issue (one major feature/fix per PR).
  • Self-Review: I reviewed my own code with care.
  • Writing Style: The PR title and description use Simplified Technical English (ASD-STE100): short sentences, plain words, active voice, one idea per sentence, no contractions.
  • Testing: New and/or updated tests have been added to cover my changes (if applicable).
  • Test Helpers (test changes only): Shared test helpers (src/test-utils/, @/utils/test-utils) are used only for mechanical duplication; test intent and scenario-specific mocks remain visible.
  • Visual Snapshot (UI changes only): N/A. No UI changes.
  • Documentation Impact: I checked if my changes need documentation updates (see "Documentation Updates" section below).
  • Contribution Guidelines: I read and agree to the Contributor Guidelines.

Visual Snapshots

N/A. No UI changes.

Videos (interaction / animation only)

N/A.

Documentation Updates

  • No documentation updates are required.

Additional Notes

  • Why Option B and not Option A: Option A must remove captures from every provider and add new capture points for mid-stream errors and every completePrompt call site. The retry loop has no bounded final failure under auto-approval. Option B removes no telemetry and keeps every capture site in place.
  • Trade-off: two distinct failures with the same fingerprint inside the 30-second window collapse into one event. This matches the issue goal of one $exception per distinct occurrence and follows the PostHog guidance on deduplication in retry loops. An identical error re-captures once a quiet gap exceeds the window.
  • The captured flag reads as undefined until the first capture, so untouched error instances stay unchanged.

Get in Touch

Discord: 0x7777777_.

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one): 1

  • coderabbit-review-active

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@zoomote

zoomote Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

No new code issues found. See session

  • src/api/providers/openrouter.ts:221-223 — Throwing the captured ApiProviderError lets PostHogTelemetryClient#captureException synchronously rewrite error.message (numeric status triggers the isOpenAISdkError branch) before the throw runs, so caller-facing text loses the OpenRouter API Error <code>: prefix whenever telemetry is enabled and the error is not 429-filtered; production PostHog payload also loses the prefix, and the spec assertions can't catch it because the telemetry module is mocked.
  • packages/telemetry/src/TelemetryService.ts:39 — The 30s dedupe window is shorter than the retry loop's exponential backoff gaps (5s doubling to a 600s cap), so from the 4th retry onward each attempt re-captures; a persistent error under auto-approval still emits roughly one $exception per backoff interval instead of one total.
  • packages/telemetry/src/PostHogTelemetryClient.ts:121 — The detached copy uses new Error() and loses the source prototype, so built-in subclasses such as TypeError are reported to PostHog as Error, and AggregateError nested errors are dropped from the payload.

Reviewed 17181aa

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Review status

This PR was opened by an automated account. A human maintainer must verify the change intent, provenance, and validation before merging.

Current step: Mark the PR ready. Required CI must pass before CodeRabbit starts.

Review-state labels are managed by this workflow; do not edit them manually. community-approved is managed the same way — do not add or remove it manually. It signals a fresh community code approval for the current head as an advisory priority only; maintainer review is still required.

@codecov

codecov Bot commented Nov 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Comment thread src/api/providers/openrouter.ts Resolved
@edelauna
edelauna force-pushed the fix/telemetry-single-capture-513 branch from 2b1b074 to 5a48226 Compare November 20, 2026 04:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] PostHog telemetry: exception inflation from double-capture and control-flow misuse of captureException

2 participants