Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 79 additions & 1 deletion src/utils/__tests__/analytics.test.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
import { Analytics, groupsFromUser } from '@utils/analytics';
import { PostHog } from 'posthog-node';
import { AxiosError } from 'axios';
import { v4 as uuidv4 } from 'uuid';
import { ANALYTICS_TEAM_TAG, WIZARD_FLAG_KEYS } from '@lib/constants';
import { VERSION } from '@lib/version';
import type { ApiUser } from '@lib/api';
import { handleApiError, type ApiUser } from '@lib/api';

vi.mock('posthog-node');
vi.mock('uuid');
Expand Down Expand Up @@ -226,6 +227,83 @@ describe('Analytics', () => {
},
);
});

it('drops a raw socket error carrying a transport errno code', () => {
const error = Object.assign(new Error('read ECONNRESET'), {
code: 'ECONNRESET',
});

analytics.captureException(error);

expect(mockPostHogInstance.captureException).not.toHaveBeenCalled();
});

it('drops a host-unreachable socket error', () => {
const error = Object.assign(
new Error('connect EHOSTUNREACH 1.2.3.4:443'),
{
code: 'EHOSTUNREACH',
},
);

analytics.captureException(error);

expect(mockPostHogInstance.captureException).not.toHaveBeenCalled();
});

it('drops a wrapped API error that folds the errno into its message', () => {
// api.ts drops `code` and leaves the errno only in the message text.
const error = new Error('Failed to fetch user data (ECONNRESET)');

analytics.captureException(error);

expect(mockPostHogInstance.captureException).not.toHaveBeenCalled();
});

it('still captures an install failure whose embedded CLI stderr mentions an errno', () => {
// A wrapped tool failure that merely quotes a benign errno in its stderr
// must still report — the errno is not the "(ECONNRESET)" wrapper api.ts
// emits, so it does not mean the user's own transport dropped.
const error = new Error(
'Codex MCP add failed: request failed ECONNRESET, retrying\npermission denied',
);

analytics.captureException(error);

expect(mockPostHogInstance.captureException).toHaveBeenCalledTimes(1);
});

it('drops an ENOTFOUND ApiError produced by handleApiError (DNS lookup failure)', () => {
// A user with no working DNS. api.ts folds the errno into the message and
// drops `code`, so the "(ENOTFOUND)" wrapper is the only trace — the same
// path the message scan handles. Must not open an error tracking issue.
const axiosError = new AxiosError('connect error');
axiosError.config = { url: '/api/users/@me/' } as never;
axiosError.code = 'ENOTFOUND';
const apiError = handleApiError(axiosError, 'fetch user data');

analytics.captureException(apiError);

expect(mockPostHogInstance.captureException).not.toHaveBeenCalled();
});

it('drops a filesystem timeout on a network-backed mount', () => {
const error = Object.assign(new Error('ETIMEDOUT: operation timed out'), {
code: 'ETIMEDOUT',
});

analytics.captureException(error);

expect(mockPostHogInstance.captureException).not.toHaveBeenCalled();
});

it('still captures a genuine wizard error', () => {
const error = new Error('Something the wizard did wrong');

analytics.captureException(error);

expect(mockPostHogInstance.captureException).toHaveBeenCalledTimes(1);
});
});

describe('flag exposure', () => {
Expand Down
60 changes: 60 additions & 0 deletions src/utils/analytics.ts
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,49 @@ export function groupsFromUser(
return groups;
}

/**
* Transport-level errno codes that mean the user's own network or machine
* dropped a connection mid-call, not that the wizard is broken. Every caller
* that hits these already degrades on its own — the Slack poll falls back to
* the connect nudge, a project-tree walk skips the entry, an API caller
* retries — so a capture adds only noise. And because each errno (and the
* host string Node folds into a raw socket message) fingerprints as its own
* error tracking issue, every one-off opens a fresh issue that buries real
* wizard bugs. Mirrors BENIGN_FS_ERROR_CODES in bounded-fs.ts.
*/
const BENIGN_TRANSPORT_ERROR_CODES: ReadonlySet<string> = new Set([
'ECONNRESET', // connection reset by peer / socket dropped
'ECONNREFUSED', // nothing listening at the far end
'ETIMEDOUT', // connection or network-backed filesystem read timed out
'EHOSTUNREACH', // no route to host
'ENETUNREACH', // no route to network
'ENETDOWN', // local network interface down
'EPIPE', // wrote to a closed socket
'EAI_AGAIN', // temporary DNS resolution failure
'ENOTFOUND', // DNS lookup failed — host not found (offline / captive portal)
]);
Comment thread
posthog[bot] marked this conversation as resolved.

/**
* The benign transport errno for an error, or undefined. Reads the `code`
* field first (raw socket and filesystem errors carry it), then falls back to
* the message — api.ts folds the errno into the ApiError message and drops
* `code`, so the parenthesized "(ECONNRESET)" wrapper is the only trace left.
* The fallback matches only that wrapper, never a bare mention: several callers
* wrap raw CLI stderr in a `new Error(...)` when an install fails, and that
* output can quote a benign errno (a "retrying ECONNRESET" log line) while the
* command actually failed for an unrelated reason. A bare substring match would
* silently drop those install failures — a class the team wants to see.
*/
function benignTransportCode(error: unknown): string | undefined {
const code = (error as NodeJS.ErrnoException | null)?.code;
if (code && BENIGN_TRANSPORT_ERROR_CODES.has(code)) return code;
const message = error instanceof Error ? error.message : '';
for (const candidate of BENIGN_TRANSPORT_ERROR_CODES) {
if (message.includes(`(${candidate})`)) return candidate;
}
Comment thread
posthog[bot] marked this conversation as resolved.
return undefined;
}

const WIZARD_FLAGS: ReadonlySet<string> = new Set(WIZARD_FLAG_KEYS);

// Widen back to the SDK's shape — a filter on `true` never matches `'true'`.
Expand Down Expand Up @@ -240,6 +283,23 @@ export class Analytics {
}

captureException(error: Error, properties: Record<string, unknown> = {}) {
// Drop transport-level failures on the user's side. They never mean the
// wizard is broken and each variant opens its own error tracking issue.
const benign = benignTransportCode(error);
if (benign) {
// This debug line is the only record of a dropped failure, and the same
// errno can come from unrelated operations (a Slack poll, a project-tree
// read, a doctor fetch). Keep the operation context (step/source) and the
// message so support can name what failed — callers already redact
// secrets from these before reporting. Mirrors bounded-fs's skip log.
const op = properties.step ?? properties.source;
logToFile(
`[analytics] skipped benign transport error (${benign})${
op ? ` [${String(op)}]` : ''
}: ${error.message}`,
);
return;
Comment thread
posthog[bot] marked this conversation as resolved.
}
Comment on lines +286 to +302

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Global filter hides fatal service failures

should_fix bug

Why we think it's a valid issue
  • Checked: every non-test caller of captureException/captureUnknown, the error shape that handleApiError builds, and the fatal-exit path in src/lib/runners/run-wizard.ts.
  • Found: handleApiError at src/lib/api.ts:298-302 returns new ApiError(\Failed to ${operation} (${axiosError.code ?? 'network error'})`)for every transport failure. The errno therefore sits in the message of each API error, so the message scan atsrc/utils/analytics.ts:101-104` drops it.
  • Found: Three capture sites report and then rethrow, so the failure is fatal and not recovered: src/utils/setup-utils.ts:672 (step: 'wizard_login', then throw error), src/utils/setup-utils.ts:552 and 585 (CI project lookup, then rethrow), and src/lib/programs/posthog-doctor/fetch.ts:24 (throw apiError). This contradicts the new comment at src/utils/analytics.ts:73-76, which states that every caller of these codes already degrades on its own.
  • Found: No other signal replaces the dropped event on the login path. The fatal catch at src/lib/runners/run-wizard.ts:249-281 logs to file, prints to the console, and calls process.exit(1). It never captures, and it never calls analytics.shutdown, which is the only source of the setup wizard finished terminal event (src/utils/analytics.ts:399-421, called from wizard-abort.ts:80, linear.ts:301, orchestrator-runner.ts:1099, start-tui.ts:90). The single captureUnknown at setup-utils.ts:672 was the only report for a failed login.
  • Found: The codebase already uses call-site filtering for the equivalent case. src/utils/bounded-fs.ts:41-44 drops benign filesystem codes inside its own reporter, where the caller is known to continue. This is the precedent the suggestion asks for.
  • Impact: If the PostHog API host becomes unreachable, or a build points at a wrong host or port, every run dies at login with ECONNREFUSED, ECONNRESET, or ETIMEDOUT, and Error Tracking records nothing at all. The team keeps no exception event and no terminal event for that run. This is a swallowed error that hides a failure, and the trigger and the consequence are both concrete.
Issue description

captureException reports both recoverable and fatal failures. Login, CI, and health-check paths capture errors before they rethrow them. A reset, refusal, or timeout now returns here without an error event. An errno cannot show whether the user, PostHog, or the wizard caused the failure. A service outage can therefore stop the wizard while Error Tracking stays silent.

Suggested fix

Make suppression opt-in for callers that recover. For example, add an option and call captureException(error, properties, { suppressTransportErrors: true }) from the Slack poll and filesystem reporter. Keep the default capture path for fatal failures. Add tests for both recovered and rethrown transport errors.

Prompt to fix with AI (copy-paste)
## Context
@src/utils/analytics.ts#L280-286

<issue_description>
`captureException` reports both recoverable and fatal failures. Login, CI, and health-check paths capture errors before they rethrow them. A reset, refusal, or timeout now returns here without an error event. An errno cannot show whether the user, PostHog, or the wizard caused the failure. A service outage can therefore stop the wizard while Error Tracking stays silent.
</issue_description>

<issue_validation>
- **Checked:** every non-test caller of `captureException`/`captureUnknown`, the error shape that `handleApiError` builds, and the fatal-exit path in `src/lib/runners/run-wizard.ts`.
- **Found:** `handleApiError` at `src/lib/api.ts:298-302` returns `new ApiError(\`Failed to ${operation} (${axiosError.code ?? 'network error'})\`)` for every transport failure. The errno therefore sits in the message of each API error, so the message scan at `src/utils/analytics.ts:101-104` drops it.
- **Found:** Three capture sites report and then rethrow, so the failure is fatal and not recovered: `src/utils/setup-utils.ts:672` (`step: 'wizard_login'`, then `throw error`), `src/utils/setup-utils.ts:552` and `585` (CI project lookup, then rethrow), and `src/lib/programs/posthog-doctor/fetch.ts:24` (`throw apiError`). This contradicts the new comment at `src/utils/analytics.ts:73-76`, which states that every caller of these codes already degrades on its own.
- **Found:** No other signal replaces the dropped event on the login path. The fatal catch at `src/lib/runners/run-wizard.ts:249-281` logs to file, prints to the console, and calls `process.exit(1)`. It never captures, and it never calls `analytics.shutdown`, which is the only source of the `setup wizard finished` terminal event (`src/utils/analytics.ts:399-421`, called from `wizard-abort.ts:80`, `linear.ts:301`, `orchestrator-runner.ts:1099`, `start-tui.ts:90`). The single `captureUnknown` at `setup-utils.ts:672` was the only report for a failed login.
- **Found:** The codebase already uses call-site filtering for the equivalent case. `src/utils/bounded-fs.ts:41-44` drops benign filesystem codes inside its own reporter, where the caller is known to continue. This is the precedent the suggestion asks for.
- **Impact:** If the PostHog API host becomes unreachable, or a build points at a wrong host or port, every run dies at login with `ECONNREFUSED`, `ECONNRESET`, or `ETIMEDOUT`, and Error Tracking records nothing at all. The team keeps no exception event and no terminal event for that run. This is a swallowed error that hides a failure, and the trigger and the consequence are both concrete.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Make suppression opt-in for callers that recover. For example, add an option and call `captureException(error, properties, { suppressTransportErrors: true })` from the Slack poll and filesystem reporter. Keep the default capture path for fatal failures. Add tests for both recovered and rethrown transport errors.
</potential_solution>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — this is a real gap at current head, not a stale flag. I traced every capture site: the global filter in captureException drops fatal transport failures, not just recovered ones.

  • The CI project lookup (fetchProjectDataWithApiKey/fetchProjectDataById in setup-utils.ts) and the doctor health-issues fetch (posthog-doctor/fetch.ts) both capture and then rethrow. handleApiError folds the errno into the message (Failed to … (ECONNRESET)) and drops code, so the message scan matches and the event is dropped before it's ever sent.
  • The rethrow lands on the fatal exit in run-wizard.ts, which logs to file and process.exit(1) without capturing and without calling analytics.shutdown — so neither an exception event nor a terminal event survives.

Net effect: if a build points at a wrong host/port, or the API times out, a run can die at login/CI/doctor with nothing in Error Tracking. The PR comment's premise that 'every caller already degrades on its own' doesn't hold for these rethrowing sites.

I'm not fixing this unattended because the fix is a design decision. The suggested opt-in suppressTransportErrors flag reverses this PR's central choice — 'quiet once at the capture site, not at each call site' — and adds a new option to the analytics API. There are a few defensible resolutions: (a) opt-in suppression at the two recovering callers (Slack poll + bounded-fs) while fatal paths keep capturing; (b) move filtering entirely back to those recovering sites and drop the chokepoint; or (c) accept the tradeoff, on the view that a true outage can't transmit a capture anyway and only the misconfigured-host case genuinely loses signal.

Human decision needed: keep the chokepoint filter and accept that fatal login/CI/doctor transport failures stop capturing, or make suppression opt-in so those fatal paths capture again. I'll implement whichever you choose.

this.client.captureException(error, this.distinctId ?? this.anonymousId, {
team: ANALYTICS_TEAM_TAG,
...this.tags,
Expand Down