test(stack): fix 1 flaky test in the Realtime catalog check by avallete · Pull Request #7359 · supabase/cli

avallete · 2026-10-28T07:41:14Z

TL;DR

The Realtime, pg-meta and Studio catalog test hung on Podman without saying where. It now fails before the test timeout, naming the service and phase that stalled and that service's output.

Realtime.integration.test.ts › serves Realtime, pg-meta, and Studio against the owned database: on Podman, the test hit the 120s test timeout with only the generator start frame as a clue. The test runs four services in sequence (database, Realtime, pg-meta, Studio), each with a recipe, start, readiness and HTTP step, so a stall anywhere looked the same. The cause is still unknown, so this change is a diagnostic. The scoped body now runs under a 110s guard. Each step records its phase, and each recipe's launch output is collected. A stall fails with CatalogStallError naming the service and phase in progress (including stop and root cleanup) and the last part of that service's output.

ref

spotted in:

https://github.com/supabase/cli/actions/runs/40530528881/job/122281757547

Generated by Claude Code

The Realtime, pg-meta and Studio catalog test hit the 120s test timeout on Podman with only the generator frame to go on. It now runs under a 110s guard that fails with CatalogStallError naming the service and phase in progress, plus that service's launch output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBEk1PCdaXDCUPR16M9Ria

avallete · 2026-10-28T07:41:20Z

/ai-review

Generated by Claude Code

github-actions

Superseded by a newer AI review

🤖 AI Review

Verified all four reported findings and merged the duplicate timeout reports. Confirmed one minor cleanup-diagnostic issue and two diagnostic nits. Updated stale line references to match the checked-out code.

Findings

Severity	Location	Category	Sources	Claim
🟡 MINOR	`packages/stack/src/services/Realtime.integration.test.ts:211`	`test-reliability`	claude+codex	The timeout guard waits for scoped finalization, so a hanging teardown can prevent CatalogStallError from reaching the test runner before its timeout.
⚪ NIT	`packages/stack/src/services/Realtime.integration.test.ts:209`	`diagnostics`	claude	Some phase labels identify a narrower operation than the work actually running, which can misdirect stall diagnosis.
⚪ NIT	`packages/stack/src/services/Realtime.integration.test.ts:43`	`correctness`	claude	Decoding each output chunk independently can corrupt split UTF-8 characters, and trimming by UTF-16 code units can leave a dangling surrogate.

Stats

Claude findings: 3 · Codex findings: 1 · Confirmed: 3 · Refuted: 0 · Uncertain: 0

Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: manual · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

github-actions

🤖 AI Review

Merged four findings into three. Confirmed the phase-label and UTF-8 decoding nits. The cleanup-timeout concern remains uncertain: scope placement and unbounded cleanup are verified, but the pinned Effect implementation is unavailable, and the patched Vitest runner complicates the claimed diagnostic loss.

Findings

Severity	Location	Category	Sources	Claim
🟡 MINOR	`packages/stack/src/services/Realtime.integration.test.ts:211`	`test-diagnostics`	claude+codex	Stalled scope cleanup may prevent the 110-second guard from surfacing CatalogStallError before Vitest's 120-second timeout.
⚪ NIT	`packages/stack/src/services/Realtime.integration.test.ts:35`	`test-diagnostics`	claude	Filesystem root setup is labeled 'database recipe', so a stall during that setup would identify the wrong phase.
⚪ NIT	`packages/stack/src/services/Realtime.integration.test.ts:43`	`correctness`	codex	Decoding each output chunk with a fresh TextDecoder corrupts UTF-8 characters split across chunk boundaries.

Stats

Claude findings: 2 · Codex findings: 2 · Confirmed: 2 · Refuted: 0 · Uncertain: 1

Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

The guard now joins a detached body under its deadline, so a teardown that hangs during interruption cannot delay the stall report. Phase labels cover root setup, the pg-meta relay and scope teardown, and launch output is decoded as a stream per channel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBEk1PCdaXDCUPR16M9Ria

github-actions · 2026-10-28T07:48:09Z

Flaky test check: ✅ no flaky tests

Commit e9d829dcbef2 · focused ×10 (5 executions each), 1 test · integration ×10, 5635 tests · unit ×10, 8394 tests · workflow run

_{Each execution is a separate Vitest process; Where lists failing runs, as run.iteration for suites executed more than once per run. The full data is in the flaky-check-report artifact.}

github-actions · 2026-11-02T15:16:34Z

CLI benchmark (Lightweight Commands): no changes found

Commit b491c29 against develop at c3f3ef2 · 10 runs each · report only, never blocks merging

Command	Workload	Before	After	Change
`supabase --version`	`version`	333 ms	327 ms	no real change
`supabase --help`	`help`	427 ms	432 ms	no real change
`supabase init`	`init`	465 ms	463 ms	no real change
`supabase migration new bench`	`migration-new`	423 ms	427 ms	no real change
`supabase gen types --lang typescript --project-id replayfixtureprojref`	`gen-types`	585 ms	559 ms	no real change
`supabase projects list`	`projects-list`	578 ms	676 ms	no real change (+17%, within normal variation)
`supabase functions list --project-ref replayfixtureprojref`	`functions-list`	539 ms	549 ms	no real change

API calls: unchanged for all 7 commands.

Binary size: 134.0 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command	Workload	Before range	After range	Change	Verdict
`supabase --version`	`version`	271-427 ms	293-395 ms	-6 ms (-2%)	within noise
`supabase --help`	`help`	404-508 ms	375-629 ms	+4 ms (+1%)	within noise
`supabase init`	`init`	328-542 ms	409-601 ms	-2 ms (0%)	within noise
`supabase migration new bench`	`migration-new`	333-574 ms	362-610 ms	+4 ms (+1%)	within noise
`supabase gen types --lang typescript --project-id replayfixtureprojref`	`gen-types`	450-714 ms	466-692 ms	-26 ms (-4%)	within noise
`supabase projects list`	`projects-list`	401-811 ms	481-937 ms	+98 ms (+17%)	within noise
`supabase functions list --project-ref replayfixtureprojref`	`functions-list`	472-601 ms	499-753 ms	+11 ms (+2%)	within noise

API calls per command

Command	Workload	Before	After
`supabase --version`	`version`	0	0
`supabase --help`	`help`	0	0
`supabase init`	`init`	0	0
`supabase migration new bench`	`migration-new`	0	0
`supabase gen types --lang typescript --project-id replayfixtureprojref`	`gen-types`	2	2
`supabase projects list`	`projects-list`	1	1
`supabase functions list --project-ref replayfixtureprojref`	`functions-list`	2	2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Workflow run

Comment /bench to run the lightweight benchmark (this comment) again on the latest commit, or /bench full to also run the integration-heavy benchmark, which posts in its own comment.

Coly010

LGTM, thanks! small optional one: on a stall the message only carries the stalled service's output, so if Realtime readiness stalls because postgres died, the bit that explains it isn't there. might be worth appending the database output too

Realtime, pg-meta and Studio all depend on the owned database, so a stall in one of them now also reports the database's launch output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MBEk1PCdaXDCUPR16M9Ria

avallete · 2026-11-02T16:07:09Z

@Coly010 thanks, done in c15f5ba: when a non-database phase stalls, the error now includes the database's output after the stalled service's.

Generated by Claude Code

…base#7318) ## TL;DR The Mailpit launch test hung on Podman without saying where. It now fails within its timeout, naming the phase that stalled and Mailpit's output. The detached stall guard it uses is now one shared test helper, `failOnStall`, which the Realtime and native owner-logs tests also use. - `Mail.integration.test.ts › launches the real Mailpit recipe and serves its HTTP endpoint`: on Podman, the test hit the 30s test timeout with no diagnostic. The error frame pointed at `sendTestEmail`, but this test never sends mail, so that frame is misleading. The test waits on start, readiness and a `/readyz` request. Container readiness may wait up to 60s, longer than the test timeout, and the `/readyz` request has no bound. So a stall in any of the three looked the same. The whole test scope now runs under one 25s guard inside the 30s test timeout, and a `Ref` tracks the current phase. The test also collects the recipe's log stream, so a stall fails with `MailStallError` naming the phase and showing Mailpit's output so far. - `packages/stack/tests/stall-guard.ts` adds `failOnStall(duration, onStall)`. It joins the guarded effect from a detached fiber, so the stall error surfaces at the deadline instead of waiting for teardown, and it interrupts the stalled fiber in the background so scoped cleanup still runs. `Mail`, `Realtime` (supabase#7359) and `Owner.logs` (supabase#7434) integration tests now share it instead of carrying three copies; each keeps its own stall message. ## ref spotted in: - https://github.com/supabase/cli/actions/runs/39920135558/job/120303098411 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>

avallete requested a review from a team as a code owner October 28, 2026 07:41

avallete added the run-flaky-check Run the flaky-test check on this PR label Oct 28, 2026 — with Claude

github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Oct 28, 2026

github-actions Bot reviewed Oct 28, 2026

View reviewed changes

Comment thread packages/stack/src/services/Realtime.integration.test.ts Resolved

Comment thread packages/stack/src/services/Realtime.integration.test.ts Resolved

Comment thread packages/stack/src/services/Realtime.integration.test.ts Resolved

github-actions Bot reviewed Oct 28, 2026

View reviewed changes

Comment thread packages/stack/src/services/Realtime.integration.test.ts Resolved

Comment thread packages/stack/src/services/Realtime.integration.test.ts Resolved

Comment thread packages/stack/src/services/Realtime.integration.test.ts Resolved

avallete added the run-flaky-check Run the flaky-test check on this PR label Oct 28, 2026 — with Claude

github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Oct 28, 2026

Merge branch 'develop' into avallete/flaky-fixes-2026-10-28

b491c29

Coly010 mentioned this pull request Nov 2, 2026

test(stack): fix 1 flaky test in the native owner-logs check #7434

Merged

Coly010 approved these changes Nov 2, 2026

View reviewed changes

Coly010 mentioned this pull request Nov 2, 2026

test(stack): fix 1 flaky Mailpit test and share the stall guard #7318

Merged

avallete added this pull request to the merge queue Nov 2, 2026

Merged via the queue into develop with commit 0f6748a Nov 2, 2026
45 checks passed

avallete deleted the avallete/flaky-fixes-2026-10-28 branch November 2, 2026 16:42

supabase-cli-releaser Bot mentioned this pull request Nov 3, 2026

chore: production deploy #7452

Open

Navigation Menu

Repository navigation

test(stack): fix 1 flaky test in the Realtime catalog check - #7359

test(stack): fix 1 flaky test in the Realtime catalog check#7359
avallete merged 4 commits into
developfrom
avallete/flaky-fixes-2026-10-28

avallete commented Oct 28, 2026

Uh oh!

avallete commented Oct 28, 2026

Uh oh!

github-actions Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 2, 2026

Timing ranges

API calls per command

Slowest phases

Uh oh!

Coly010 left a comment

Uh oh!

avallete commented Nov 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

avallete commented Oct 28, 2026

TL;DR

ref

Uh oh!

avallete commented Oct 28, 2026

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

🤖 AI Review

Findings

Stats

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

🤖 AI Review

Findings

Stats

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Flaky test check: ✅ no flaky tests

Uh oh!

github-actions Bot commented Nov 2, 2026

CLI benchmark (Lightweight Commands): no changes found

Timing ranges

API calls per command

Slowest phases

Uh oh!

Coly010 left a comment

Choose a reason for hiding this comment

Uh oh!

avallete commented Nov 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot left a comment •

edited

Loading

github-actions Bot commented Oct 28, 2026 •

edited

Loading