Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file.
75 changes: 75 additions & 0 deletions .claude/skills/audit/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
---
name: audit
description: Audit an existing PostHog integration for correctness and best practices
metadata:
author: PostHog
version: 1.26.0
---

# PostHog Audit

This skill audits an existing PostHog integration for **data integrity** in event capture and identification. **Read-only** — the only file you create is the final audit report.

Perform the checks described in the referenced skills and only the events referenced in the skills.

## Workflow

The audit runs as a 5-step chain: Installation (SDK + version) → init correctness → identification → event capture → report (which also uploads the report to a PostHog notebook). Each step file ends with a pointer to the next. Follow them in the order they are written. You must resolve them in order before any source-tree exploration.

The audit ledger is already seeded with the 11 pending checks (10 correctness checks plus `upload-notebook`, which the report step resolves after mirroring the markdown into a PostHog notebook). Use `mcp__wizard-tools__audit_resolve_checks` to patch each one as you finish it.

**Start by reading the path relative to this file at `references/1-version.md`.** Do not Glob, ls, or find the skill directory. Do not preload future steps. Do not re-read a step file once you've moved past it. Do not re-read SKILL.md.

`ToolSearch` is only for loading a tool by exact name when the SDK has it deferred (e.g. `select:Grep`). Do **not** use it to browse for other tools — every tool the audit needs (`Glob`, `Grep`, `Read`, `Write`, `Bash`, and the named `mcp__wizard-tools__audit_*` tools) is already named in this skill.

**Do not call `TaskCreate` / `TaskUpdate` / `TaskGet` / `TaskList`.** The audit doesn't track its own task list — progress comes from the audit ledger plus `[STATUS]` lines.

## Live activity — `[STATUS]`

The "Working on …" banner reads from `[STATUS]` lines you emit in plain text. Whenever you start a new sub-step, write a line like:

```
[STATUS] Scanning manifests
```

The wizard intercepts these and updates the spinner. Use them freely — they are cheap. Each step file lists the exact `[STATUS]` strings to emit at each sub-step.

## Audit checks ledger

The ledger lives at `.posthog-audit-checks.json` and is rendered live in the "Audit plan" tab. It is owned by MCP tools — **never `Write` this file directly**:

- `mcp__wizard-tools__audit_resolve_checks({ updates })` — patch one or more checks by `id`. Each `update` is `{ id, status, file?, details? }`. Batch updates from the same step into a single call.

All audit ledger calls are atomic and serialize internally — **concurrent calls from parallel subagents cannot lose updates**, so feel free to fan out runtime checks across `Agent` subagents when a step says so.

### Check entry shape

- `id` — stable kebab-case slug. Reuse the existing seeded ids exactly when calling `audit_resolve_checks`.
- `area` — short group name. The current core workflow uses `Installation`, `Identification`, and `Event Capture`.
- `label` — short human name.
- `status` — `pending` | `pass` | `error` | `warning` | `suggestion`.
- `file` — optional `path:line` for findings tied to a location.
- `details` — optional one-line explanation.

After the report is written (Step 5), delete `.posthog-audit-checks.json`.

## Severity levels

- `error`: Must fix. Broken functionality, data corruption, or security issue.
- `warning`: Should fix. Pattern that causes subtle bugs or data-quality problems.
- `suggestion`: Nice to have. Best-practice improvement.

## Key principles

- **Read-only**: Do not edit project source files. The only file you create is the audit report.
- **Evidence-based**: Reference specific `file:line` for every non-pass finding.
- **Actionable**: Every finding states what to fix and how.

## Abort statuses

Report abort states with `[ABORT]` prefixed messages. The wizard catches these and terminates the run — do not halt yourself.
- No PostHog SDK found

## Framework guidelines

_No specific framework guidelines._
90 changes: 90 additions & 0 deletions .claude/skills/audit/references/1-version.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Step 1 — SDK installed + SDK up-to-date

**Read ONLY this file.** Do not read any other reference file until this one tells you to.

This step is intentionally narrow. It runs **before any other project work**. Resolve exactly two checks: `sdk-installed` and `sdk-up-to-date`. **Do not** read source code, locate init sites, look at `.env*` files, or scan for identify/capture call sites in this step — that all belongs to later steps.

## Status

Emit:

```
[STATUS] Scanning manifests
[STATUS] Checking SDK version
```

## Action

### a. Find the PostHog SDK

`Glob` for the project's dependency manifests across every language PostHog ships an SDK for. The full list:

- `package.json` — npm / pnpm / yarn (Node, web, React, Next.js, Nuxt, Vue, Svelte, Angular, React Native, Expo)
- `requirements.txt`, `pyproject.toml`, `Pipfile`, `setup.py` — Python (Django, Flask, FastAPI, etc.)
- `Gemfile` — Ruby / Ruby on Rails
- `composer.json` — PHP / Laravel
- `go.mod` — Go
- `build.gradle`, `build.gradle.kts`, `pom.xml` — Java / Android
- `Podfile`, `Package.swift` — iOS / Swift
- `pubspec.yaml` — Flutter / Dart
- `*.csproj` — .NET
- `mix.exs` — Elixir

Read enough of them to identify which PostHog SDK the project uses, what version, and what framework it sits on top of.

If no PostHog SDK is anywhere in the project, emit `[ABORT] No PostHog SDK found` and stop. The wizard catches `[ABORT]` and terminates the run.

### b. Install the matching integration skill

Once you know the SDK + framework, install the matching integration skill so the rest of the audit has framework-specific install docs to reference instead of guessing:

1. Call `mcp__wizard-tools__load_skill_menu({ category: "integration" })` once to list available integration skill IDs.
2. Call `mcp__wizard-tools__install_skill({ skillId: "<id>" })` with the **single** ID that matches the framework you detected. Pick one — do not install multiple.

If no integration skill matches the framework, skip this step. Step 2 will fall back to general framework knowledge.

### c. Check latest published version

For each detected SDK, run `Bash` once to look up the latest published version. Use the command that matches the SDK's registry:

- **npm** (JS/TS, Node, React, Next.js, Nuxt, Vue, Svelte, Angular, React Native, Expo): `npm view <pkg> version`
- **PyPI** (Python): `pip index versions <pkg>` (or `pip show <pkg>` if `index` is unavailable)
- **RubyGems** (Ruby / Rails): `gem search ^<pkg>$ -r`
- **Packagist** (PHP / Laravel): `composer show <pkg> --latest --available --format=json`
- **Go modules** (Go): `curl -s https://proxy.golang.org/<module>/@latest` (returns JSON with the latest `Version`)
- **Maven Central** (Java / Android): `curl -s "https://search.maven.org/solrsearch/select?q=g:<group>+AND+a:<artifact>&rows=1&wt=json"` and read `.response.docs[0].latestVersion`
- **CocoaPods** (iOS / Swift): `pod search <pkg>` (or check `https://cdn.cocoapods.org/all_pods_versions_<x>_<y>_<z>.txt` for the spec mirror)
- **Swift Package Manager** (Swift): `gh release list --repo posthog/posthog-ios --limit 1` (SwiftPM resolves from GitHub tags)
- **pub.dev** (Flutter / Dart): `curl -s https://pub.dev/api/packages/<pkg> | jq -r .latest.version`
- **NuGet** (.NET): `curl -s https://api.nuget.org/v3-flatcontainer/<pkg>/index.json | jq -r '.versions[-1]'`
- **Hex** (Elixir): `mix hex.info <pkg>`

## Resolution rules

`sdk-installed`:
- `pass`: at least one PostHog SDK in a manifest. Record SDK + version in `details`.

`sdk-up-to-date`:
- `pass`: at the latest minor.
- `suggestion`: patch-only behind.
- `warning`: more than one minor behind.
- `error`: one or more major versions behind.

## Resolve

Single call to `mcp__wizard-tools__audit_resolve_checks` with two updates and **nothing else**:

```
{
"updates": [
{ "id": "sdk-installed", "status": "pass", "details": "<sdk>@<version>" },
{ "id": "sdk-up-to-date", "status": "pass|suggestion|warning|error", "details": "installed <v>, latest <v>" }
]
}
```

Do not include `init-correct` in this call — it's resolved in Step 2.

---

**Upon completion, continue with:** [2-init.md](2-init.md)
52 changes: 52 additions & 0 deletions .claude/skills/audit/references/2-init.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Step 2 — Init correctness

**Read ONLY this file.** Do not read any other reference file until this one tells you to.

This step resolves two checks: `init-correct` and `init-not-duplicated`. Manifests and SDK versions are already resolved (Step 1). Identification call sites belong to Step 3 and event-capture call sites to Step 4 — do not scan for them here.

## Status

Emit:

```
[STATUS] Locating PostHog initialization
```

## Action

Locate every PostHog init site by issuing one `Grep` for `posthog\.init\(|new PostHog\(|posthog\.Posthog\(|Posthog\(` plus whatever `Read` calls are needed. Confirm at least one init exists, runs in the right runtime for the detected SDK + framework, and sources its token from an env variable (not hardcoded). Also check `.env*` files to confirm the token env var is actually set. Reverse-proxy / `api_host` configuration belongs to Step 4 — don't evaluate it here.

Use the detected SDK + framework from Step 1 to know what to look for: the canonical init filename, runtime, and shape vary by framework. If the host project already ships a PostHog integration skill, use that as the source of truth. Skills are typically under `.claude/skills/`; if that directory doesn't exist (some projects keep skills under `agents/skills/`, plain `skills/`, etc.), discover any candidates with one `Glob` pattern: `**/skills/**/SKILL.md`. Read the matching skill before judging.

When no integration skill is available, rely on general framework knowledge — and stay conservative on `init-correct` (prefer `warning` over `error` when the convention is unclear).

For `init-not-duplicated`, count the init sites the grep found and group them by runtime (browser vs. server vs. mobile). The check fires when more than one init site exists in the same runtime — the browser bundle running `posthog.init()` twice will race to stamp `$set_once` properties like `$initial_pathname` and `$initial_referrer`, corrupting attribution at scale. A browser init plus a server init is fine; two browser inits is not. Treat conditional inits inside an `if (typeof window === 'undefined')` / `if (typeof window !== 'undefined')` branch on the same code path as ONE logical site per runtime (only one branch executes).

## Resolution rules

`init-correct`:
- `pass`: at least one init present, env-sourced token, runtime-appropriate location.
- `error`: init missing, hardcoded token, or wrong runtime (e.g. server-only init for a browser-side framework).
- `warning`: init present but in a non-canonical location for the framework.

`init-not-duplicated`:
- `pass`: at most one init site per runtime.
- `warning`: two or more init sites in the same runtime that could plausibly both execute. Report the file:line of each and which runtime. The dual-init `$set_once` race corrupted attribution at scale at one multi-SDK SaaS customer (~165k blocked merges/day).
- `error`: two or more browser-runtime init sites confirmed to run on the same page (e.g. one in the root layout / provider, another in a child component that always mounts).

## Resolve

Single call to `mcp__wizard-tools__audit_resolve_checks` with two updates:

```
{
"updates": [
{ "id": "init-correct", "status": "pass|error|warning", "file": "<path:line>", "details": "..." },
{ "id": "init-not-duplicated", "status": "pass|warning|error", "file": "<path:line>", "details": "..." }
]
}
```

---

**Upon completion, continue with:** [3-identification.md](3-identification.md)
118 changes: 118 additions & 0 deletions .claude/skills/audit/references/3-identification.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# Step 3 — Identification

**Read ONLY this file.** Do not read any other reference file until this one tells you to.

This step resolves four identification checks **in parallel**, one subagent per check:

- `identify-stable-distinct-id`
- `identify-not-late`
- `cross-runtime-distinct-id`
- `identify-reset-on-logout`

Each subagent owns its own grep, reads, evaluates its single rule, and emits one `audit_resolve_checks` call with one update. The ledger's mutex serializes concurrent writes — there's no race.

## Status

Emit before dispatching:

```
[STATUS] Auditing identification
```

## Action — dispatch four subagents in one message

Make **four `Agent` tool calls in a single message** so they run concurrently. Wait for all four to return, then continue to `4-event-capture.md`. Do not run any other tools between dispatch and the next step.

The bundled `identify-users.md` reference holds PostHog's authoritative guidance on `distinct_id`, `identify()` ordering, and cross-runtime identity. It's typically at `.claude/skills/audit/references/identify-users.md`; if that path doesn't exist, discover it with `Glob` `**/skills/audit/references/identify-users.md`. Each subagent reads it once before judging.

### Task A — `identify-stable-distinct-id`

`description`: `Audit identify-stable-distinct-id`

`prompt`:
```
You are an audit subagent. Resolve exactly one rule and return: identify-stable-distinct-id.

Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit/references/identify-users.md`).

Run **one** Grep: `posthog\.identify\(`. Read each file that contains a hit, once. Inspect the first argument passed to identify().

Rule:
- distinct_id must be a stable identifier (auth user id, account id), not a session UUID, ephemeral cookie, or device-only id.
- pass: sources from authenticated user (session.user.id, auth.uid(), etc.)
- error: sources from a session, request, or device id that resets
- warning: source unclear — flag for human review

Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `identify-stable-distinct-id`, including `file` (path:line) and `details` (one-line explanation). Return when the call completes. Do not write the audit report.
```

### Task B — `identify-not-late`

`description`: `Audit identify-not-late`

`prompt`:
```
You are an audit subagent. Resolve exactly one rule and return: identify-not-late.

Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit/references/identify-users.md`).

Run **two** Greps in parallel:
- `posthog\.identify\(` — where identity is established
- `posthog\.capture\(|getFeatureFlag\(|isFeatureEnabled\(` — where captures and flag evals happen

Read each file that contains a hit, once. Compare the timing/ordering of identify() against the surrounding capture / flag-eval calls.

Rule:
- identify() must be called before any posthog.capture for that user, and before any feature-flag eval depending on user identity.
- pass: identify runs at session start / right after login. Captures and flag evals come after.
- warning: identify runs lazily (e.g. settings-page mount), so early captures and flag evals are anonymous.

Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `identify-not-late`, including `file` (path:line of the identify call) and `details` (one-line explanation). Return when the call completes. Do not write the audit report.
```

### Task C — `cross-runtime-distinct-id`

`description`: `Audit cross-runtime-distinct-id`

`prompt`:
```
You are an audit subagent. Resolve exactly one rule and return: cross-runtime-distinct-id.

Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit/references/identify-users.md`).

Run **one** Grep: `posthog\.init\(|new PostHog\(|posthog\.Posthog\(|Posthog\(` — locate every PostHog initialization across runtimes. Read each file that contains a hit, once. Determine whether both client and server runtimes initialize PostHog, and if so, how distinct_id flows between them.

Rule:
- If both client and server runtimes call PostHog, the same distinct_id must be used on both sides for the same user.
- pass: server-side captures source the client's distinct_id (cookie, session token, or explicit hand-off).
- error: server-side captures use a different identifier scheme.
- Skip (`pass` with details: "single runtime"): only one runtime initializes PostHog.

Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `cross-runtime-distinct-id`, including `file` (path:line of the most relevant init or capture site) and `details` (one-line explanation). Return when the call completes. Do not write the audit report.
```

### Task D — `identify-reset-on-logout`

`description`: `Audit identify-reset-on-logout`

`prompt`:
```
You are an audit subagent. Resolve exactly one rule and return: identify-reset-on-logout.

Read this skill's bundled `identify-users.md` reference once (typically `.claude/skills/audit/references/identify-users.md`; otherwise discover with `Glob` `**/skills/audit/references/identify-users.md`).

Locate logout, sign-out, and account-switching flows by issuing whatever `Grep` and `Read` calls are needed in parallel. Determine whether those flows clear PostHog state with `posthog.reset()`.

Rule:
- Logout or account-switching flows should call `posthog.reset()`. Without a reset, when user B logs in on the same device after user A, PostHog's anonymous ID is shared and the next `identify()` can merge both accounts into one person.
- pass: every detected logout/account-switch flow calls `posthog.reset()`.
- error: a logout/account-switch flow is missing `posthog.reset()`.
- Skip (`pass` with details: "no logout/account-switch flow found"): no detectable logout/account-switch flow exists.
- note: `posthog.reset(true)` is valid when a completely clean device ID reset is required.

Emit one `mcp__wizard-tools__audit_resolve_checks` call with a single update for id `identify-reset-on-logout`, including `file` (path:line of the most relevant logout or reset site) and `details` (one-line explanation). Return when the call completes. Do not write the audit report.
```

---

**Upon completion, continue with:** [4-event-capture.md](4-event-capture.md)
Loading
Loading