Skip to content

fix(self-driving): drop replay from the step 4 native sources #1174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
skoob13 merged 2 commits into from
Sep 1, 2026
+12 −9
Merged

fix(self-driving): drop replay from the step 4 native sources #1174

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ef601d1
fix(self-driving): drop replay from the step 4 native sources
skoob13 Aug 31, 2026
b95be32
Merge branch 'main' into fix/self-driving-retired-replay-source
skoob13 Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions src/lib/programs/self-driving/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,10 @@ the number.
inert (recorded for the report). See §9.
- **4 — Enable sources** — always enable the scout gate and health checks;
enable the native sources whose products step 3b turned on (error tracking,
replay, support) by default, plus any other native source step-2 evidence
shows in use. Support's source stays idle until a channel is connected (a
follow-up).
support) by default, plus any other native source step-2 evidence shows in
use. Support's source stays idle until a channel is connected (a follow-up).
Replay has no source here — session summarization was retired server-side, so
step 6c's scanners are the replay path.
Comment on lines +91 to +94

@posthog posthog Bot Sep 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the remaining product-enablement rationale

should_fix documentation

Why we think it's a valid issue
  • Checked: the full PR diff (gh pr diff 1174), then every statement in ARCHITECTURE.md about why step 3b enables products, located with a grep for have data and unchanged.
  • Found: ARCHITECTURE.md:85-88 still reads "turn ON Session Replay + Error Tracking + Support via products-enable (server-owned recipes) so the next step's sources have data." The PR corrected exactly this sentence in the sibling file — prompt.ts:145-148 now reads "so the sources you enable next, and the Replay Vision scanners in STEP 6c, have data to read." The doc copy of the same sentence was left as it was.
  • Found: ARCHITECTURE.md:689-691 repeats the retired rationale: "step turns PostHog products ON (so the signal sources have data to read) before sources are enabled."
  • Found: ARCHITECTURE.md:704-705 states "STEP 4 is unchanged — once products are on, its existing 'enable sources for products in use' rule picks them up." Step 4 for replay is the one thing this PR changes.
  • Found: §9 is not a frozen historical record. Its own header at ARCHITECTURE.md:695-696 says "the corrections from the build are folded into 9.1/9.3/9.7/9.8", so §9.1 is a live statement a maintainer is told to trust.
  • Impact: the file now contradicts itself. ARCHITECTURE.md:93-94 and row 4 of the step table say replay has no source in step 4; lines 86, 690 and 704 say step 3b enables replay so step 4's sources have data, and that step 4 needs no change. A maintainer or an agent that reads §9.1 can re-create the retired session_replay source row, which emits nothing — the exact regression this PR exists to prevent. The doc is the deliverable here, not a side comment, so the leftover text undoes the change in the same file.
  • Impact: the fix is small and mechanical: point the replay rationale at step 6c's scanners in all three places, and drop "STEP 4 is unchanged". I confirmed no other stale reference exists — the two remaining session_replay hits (ARCHITECTURE.md:135, :771) are the products-enable product key, not the retired source type.
Issue description

The new STEP 4 summary says that replay has no native source. Lines 85-87 still say STEP 3b enables all three products for STEP 4. Lines 690-691 and 704-705 repeat this claim. These statements can lead a maintainer to restore the retired replay source.

Suggested fix

Map Error Tracking and Support to STEP 4. Map Session Replay only to STEP 6c. Also remove the claim that STEP 4 is unchanged.

Prompt to fix with AI (copy-paste) 
## Context
@src/lib/programs/self-driving/ARCHITECTURE.md#L91-94

<issue_description>
The new STEP 4 summary says that replay has no native source. Lines 85-87 still say STEP 3b enables all three products for STEP 4. Lines 690-691 and 704-705 repeat this claim. These statements can lead a maintainer to restore the retired replay source.
</issue_description>

<issue_validation>
- **Checked:** the full PR diff (`gh pr diff 1174`), then every statement in `ARCHITECTURE.md` about why step 3b enables products, located with a grep for `have data` and `unchanged`.
- **Found:** `ARCHITECTURE.md:85-88` still reads "turn ON Session Replay + Error Tracking + Support via `products-enable` (server-owned recipes) so the next step's sources have data." The PR corrected exactly this sentence in the sibling file — `prompt.ts:145-148` now reads "so the sources you enable next, and the Replay Vision scanners in STEP 6c, have data to read." The doc copy of the same sentence was left as it was.
- **Found:** `ARCHITECTURE.md:689-691` repeats the retired rationale: "step turns PostHog products ON (so the signal sources have data to read) **before** sources are enabled."
- **Found:** `ARCHITECTURE.md:704-705` states "**STEP 4 is unchanged** — once products are on, its existing 'enable sources for products in use' rule picks them up." Step 4 for replay is the one thing this PR changes.
- **Found:** §9 is not a frozen historical record. Its own header at `ARCHITECTURE.md:695-696` says "the corrections from the build are folded into 9.1/9.3/9.7/9.8", so §9.1 is a live statement a maintainer is told to trust.
- **Impact:** the file now contradicts itself. `ARCHITECTURE.md:93-94` and row 4 of the step table say replay has no source in step 4; lines 86, 690 and 704 say step 3b enables replay so step 4's sources have data, and that step 4 needs no change. A maintainer or an agent that reads §9.1 can re-create the retired `session_replay` source row, which emits nothing — the exact regression this PR exists to prevent. The doc is the deliverable here, not a side comment, so the leftover text undoes the change in the same file.
- **Impact:** the fix is small and mechanical: point the replay rationale at step 6c's scanners in all three places, and drop "STEP 4 is unchanged". I confirmed no other stale reference exists — the two remaining `session_replay` hits (`ARCHITECTURE.md:135`, `:771`) are the `products-enable` product key, not the retired source type.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Map Error Tracking and Support to STEP 4. Map Session Replay only to STEP 6c. Also remove the claim that STEP 4 is unchanged.
</potential_solution>

- **5 — Offer issue trackers** — one multi-select (GitHub Issues / Linear /
Zendesk / pganalyze). Auto-connect what the run can: GitHub Issues (pick a
repo — a single connected repo is used by default with no repo research;
Expand Down Expand Up @@ -132,7 +133,7 @@ The table below adds the skill reference and the tool/MCP surface for each.
| 2 | Read project & Signals state | `2-read-context.md` | `./posthog-setup-report.md` + `signals-scout-project-profile-get` + cheap usage probes. Prompt opt-ins are authoritative ("repo evidence rules a product IN, never OUT"). |
| 3 | Connect GitHub (REQUIRED) | `3-github.md` | `integrations-list` for `kind:"github"`; else `wizard_ask` with the one-click `integrations/authorize?kind=github` deep-link (the single link covers fresh install / link-existing / re-auth — no separate settings "re-link" path), re-verify after a manual "done". Can't → `[ABORT] github connection declined`. |
| 3b | Enable products | `3b-enable-products.md` | `products-enable {products:[session_replay,error_tracking,conversations]}` flips the product toggles (server-owned recipes, conservative defaults). Idempotent. Web also gets a posthog-js init check; backend/mobile are inert → recorded for the report. See §9. |
| 4 | Enable signal sources | `4-sources.md` | Create/enable `SignalSourceConfig` rows (`inbox-source-configs-*`). The native sources for the step-3b products (error tracking, replay, support) go on by default; others follow step-2 evidence. Always enables the scout gate `signals_scout`/`cross_source_issue`. Always enable the health check gate `health_checks`/`health_issue`. Never enables an unconfirmed connected tool. |
| 4 | Enable signal sources | `4-sources.md` | Create/enable `SignalSourceConfig` rows (`inbox-source-configs-*`). The native sources for the step-3b products (error tracking, support) go on by default; others follow step-2 evidence. Always enables the scout gate `signals_scout`/`cross_source_issue`. Always enable the health check gate `health_checks`/`health_issue`. Never enables an unconfirmed connected tool. |
| 5 | Offer issue-tracker integrations | `5-connected-tools.md` (+ `5a`, `5b`) | One batched multi-select for GitHub Issues / Linear / Zendesk / pganalyze. GitHub Issues & Linear auto-connect via `external-data-sources-create` (GitHub Issues: one connected repo → use it by default, no repo research; Linear: OAuth link + one silent `integrations-list`, never nudge); Zendesk / pganalyze are armed dormant + report follow-up (no UI redirect, no verify). Enable a (possibly dormant) responder per pick. |
| 6 | Configure the scout troop | `6-scouts.md` | `signals-scout-config-sync` materializes the troop (~19 scouts, grows over time); `scout-metadata-get` reports the enforced run budget (100 runs/day default); enable `general` + the **3–5 specialists** for the most-used products (agent judgment over step-2 evidence), keeping the whole troop at or under **~10 enabled scouts**, never `error-tracking`/`session-replay` (covered by native sources), fall back to one universal cross-product scout if no surface qualifies, disable all the rest (`signals-scout-config-update {enabled:false}`). Never touches `emit`/`run_interval`. |
| 6b | Design custom scouts | `6b-tailor-scouts.md` | The **only** place custom scouts are created. Gap-analyze repo surfaces vs the troop, reading the repo's for-agents context first (AGENTS.md, CLAUDE.md, ARCHITECTURE.md, `.cursor/rules`) as the map of surfaces and vocabulary; propose **at most 5** in ONE `wizard_ask` (bounded by the ~10-scout troop ceiling and the enforced run budget), each option carrying a `description` (an optional `wizard_ask` option field rendered dimmed/wrapped under the label) plus a leading "None" option that's the default highlight (so an empty submit declines); create approved ones via `llma-skill-create` (`signals-scout-<scope>`). **Canonical bodies never edited.** Declining is valid, not an abort. |
Expand Down Expand Up @@ -307,8 +308,9 @@ the surface-specific scouts (`product-analytics`, `web-analytics`,
`replay-vision`) plus the cross-product
`anomaly-detection`/`observability-gaps`/`health-checks`/`inbox-validation` —
**excluding** `error-tracking`/`session-replay`, which are deliberately never
enabled because step 4 consumes them as native sources (a scout would duplicate
that pipeline). If no surface clearly qualifies, one universal cross-product
enabled because each surface already has its own pipeline (step 4's native
source for error tracking, step 6c's scanners for replay) that a scout would
Comment on lines +311 to +312

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the remaining replay scout rationale

Apply this scanner-based rationale to every STEP 6 description: the backbone at line 109 and the table at line 138 still say that session-replay is excluded because it is consumed or covered by native sources, directly contradicting the new statement that replay has no native source. The completed-work note around lines 534–543 repeats the obsolete claim as well. Leaving these active instructions inconsistent can lead the next context-mill change to restore the retired replay source.

Useful? React with 👍 / 👎.

duplicate. If no surface clearly qualifies, one universal cross-product
scout (`anomaly-detection` or `health-checks`) is the fallback so ≥1 specialist
always runs. Everything else is disabled; the enabled troop caps at **~10**
(general
Expand Down
7 changes: 4 additions & 3 deletions src/lib/programs/self-driving/prompt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,9 +144,10 @@ STEP 3 — Connect GitHub. REQUIRED. (skill: "Connect GitHub")

STEP 3b — Enable products. (skill: "Enable products")
Turn ON the PostHog products Signals reads from — Session Replay,
Error Tracking, and Support — so the sources you enable next have data
to read. These are server-side enables with conservative defaults
(owned by the server, not you). The project-state block above covers
Error Tracking, and Support — so the sources you enable next, and the
Replay Vision scanners in STEP 6c, have data to read. These are
server-side enables with conservative defaults (owned by the server,
not you). The project-state block above covers
only Session Replay and Error Tracking, so you can skip those if they
are already ON; it does not show Support, but every enable is idempotent
so enabling any of them again is safe regardless. For a web app, also
Expand Down
Loading