Skip to content

test(stack): fix 1 flaky Mailpit test and share the stall guard - #7318

Merged
avallete merged 6 commits into
developfrom
avallete/flaky-fixes-2026-10-24
Nov 2, 2026
Merged

avallete merged 6 commits into
developfrom
avallete/flaky-fixes-2026-10-24

Conversation

@avallete

@avallete avallete commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

TL;DR

The Mailpit launch test hung on Podman without saying where. It now fails within its timeout, naming the phase that stalled and Mailpit's output. The detached stall guard it uses is now one shared test helper, failOnStall, which the Realtime and native owner-logs tests also use.

  • Mail.integration.test.ts › launches the real Mailpit recipe and serves its HTTP endpoint: on Podman, the test hit the 30s test timeout with no diagnostic. The error frame pointed at sendTestEmail, but this test never sends mail, so that frame is misleading. The test waits on start, readiness and a /readyz request. Container readiness may wait up to 60s, longer than the test timeout, and the /readyz request has no bound. So a stall in any of the three looked the same. The whole test scope now runs under one 25s guard inside the 30s test timeout, and a Ref tracks the current phase. The test also collects the recipe's log stream, so a stall fails with MailStallError naming the phase and showing Mailpit's output so far.
  • packages/stack/tests/stall-guard.ts adds failOnStall(duration, onStall). It joins the guarded effect from a detached fiber, so the stall error surfaces at the deadline instead of waiting for teardown, and it interrupts the stalled fiber in the background so scoped cleanup still runs. Mail, Realtime (test(stack): fix 1 flaky test in the Realtime catalog check #7359) and Owner.logs (test(stack): fix 1 flaky test in the native owner-logs check #7434) integration tests now share it instead of carrying three copies; each keeps its own stall message.

ref

spotted in:

🤖 Generated with Claude Code

The launch test now bounds start, readiness and the /readyz request inside
the test timeout and fails with the phase and Mailpit's output so far,
instead of a bare test timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EMySvvbM4B1zdmhw3pp1wA
@avallete
avallete requested a review from a team as a code owner October 24, 2026 07:43

Copy link
Copy Markdown
Member Author

/ai-review


Generated by Claude Code

@avallete avallete added the run-flaky-check Run the flaky-test check on this PR label Oct 24, 2026 — with Claude
@github-actions github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Oct 24, 2026
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Flaky test check: ✅ no flaky tests

Commit fc788cff4e43 · focused ×10 (5 executions each), 6 tests · integration ×10, 6682 tests · unit ×10, 9346 tests · workflow run

Each execution is a separate Vitest process; Where lists failing runs, as run.iteration for suites executed more than once per run. The full data is in the flaky-check-report artifact.

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by a newer AI review

🤖 AI Review

Claude reported one potential test-flakiness issue; Codex reported none. The reduced readiness budget is verified, but whether it causes otherwise successful Mailpit runs to fail remains uncertain. Tests were not run.

Findings

Severity Location Category Sources Claim
🟡 MINOR packages/stack/src/services/Mail.integration.test.ts:107 test-flakiness claude The fixed six-second readiness timeout can reject slow Mailpit startups that would previously have completed within the overall 30-second test timeout, potentially increasing flakiness on Podman.

Stats

Claude findings: 1 · Codex findings: 0 · Confirmed: 0 · Refuted: 0 · Uncertain: 1


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

Both independent reviews completed. Confirmed Claude’s readiness-timeout concern and UTF-8 diagnostic decoding defect. Codex reported no findings. Neither issue changes production behavior.

Findings

Severity Location Category Sources Claim
🟡 MINOR packages/stack/src/services/Mail.integration.test.ts:107 test-flakiness claude The six-second readiness timeout can fail a healthy but slower container even when most of the test’s 30-second budget remains.
🟡 MINOR packages/stack/src/services/Mail.integration.test.ts:98 correctness claude Decoding every log chunk with a fresh TextDecoder corrupts UTF-8 characters split across chunks in the timeout diagnostics.

Stats

Claude findings: 2 · Codex findings: 0 · Confirmed: 2 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: manual · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

@avallete avallete added the run-flaky-check Run the flaky-test check on this PR label Oct 24, 2026 — with Claude
@github-actions github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Oct 24, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

CLI benchmark (Lightweight Commands): no changes found

Commit ba8441c against develop at c3f3ef2 · 10 runs each · report only, never blocks merging

Command Workload Before After Change
supabase --version version 263 ms 254 ms no real change
supabase --help help 275 ms 264 ms no real change
supabase init init 340 ms 338 ms no real change
supabase migration new bench migration-new 355 ms 355 ms no real change
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 419 ms 442 ms no real change (+6%, within normal variation)
supabase projects list projects-list 387 ms 377 ms no real change
supabase functions list --project-ref replayfixtureprojref functions-list 290 ms 320 ms no real change (+10%, within normal variation)

API calls: unchanged for all 7 commands.

Binary size: 134.0 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command Workload Before range After range Change Verdict
supabase --version version 238-299 ms 229-299 ms -10 ms (-4%) within noise
supabase --help help 233-374 ms 250-336 ms -11 ms (-4%) within noise
supabase init init 269-398 ms 285-457 ms -2 ms (0%) within noise
supabase migration new bench migration-new 290-570 ms 302-437 ms +0 ms (+0%) within noise
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 305-577 ms 356-573 ms +23 ms (+6%) within noise
supabase projects list projects-list 275-440 ms 283-439 ms -10 ms (-3%) within noise
supabase functions list --project-ref replayfixtureprojref functions-list 251-381 ms 284-353 ms +30 ms (+10%) within noise

API calls per command

Command Workload Before After Endpoint changes
supabase --version version 0 0
supabase --help help 0 0
supabase init init 0 0
supabase migration new bench migration-new 0 0
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 2 2
supabase projects list projects-list 1 1
supabase functions list --project-ref replayfixtureprojref functions-list 2 2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Workflow run

Comment /bench to run the lightweight benchmark (this comment) again on the latest commit, or /bench full to also run the integration-heavy benchmark, which posts in its own comment.

@Coly010 Coly010 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one ask: the guard sits inside the Effect.scoped, so when it fires, the docker teardown (docker stop with its default 10s grace, then rm --force) runs before MailStallError surfaces. on a slow engine, which i think is the likely cause of this hang, that pushes past 30s and vitest reports the plain timeout again, with the stall error only in the logs afterwards. could this use the same detached-join shape as #7359 around the whole scope? (might be worth a shared tests/ helper at that point since #7434 has the same shape, but totally fine if you'd rather not)

approving so you're unblocked, otherwise looks good to me!

avallete commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Good catch, thanks. 6e5ecdb (not yet pushed) uses the detached-join shape from #7359 around the whole Effect.scoped. The work runs in a detached fiber, the test joins it under the 25s guard, and an interrupted join interrupts the fiber in the background. So MailStallError surfaces at the deadline, even if the docker stop and remove are still running. The phase now also covers stop and scope teardown. I checked this with a scratch test whose scope never finishes and whose finalizer sleeps 10s: the stall error arrived after about 1s. The log capture also uses streaming decoders per stream now, as #7359 does.

I left out the shared tests/ helper for now: #7359 isn't on develop yet, and pulling Realtime in would widen this PR. Once #7359 and #7434 land, the three copies can become one helper in a follow-up.


Generated by Claude Code

@avallete avallete added the run-flaky-check Run the flaky-test check on this PR label Nov 2, 2026 — with Claude
@github-actions github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Nov 2, 2026
avallete and others added 2 commits November 2, 2026 17:45
Mail, Realtime and native owner-logs now use one failOnStall helper that
fails at the deadline without waiting for the stalled scope's finalizers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@avallete avallete changed the title test(stack): fix 1 flaky test in the Mailpit launch check test(stack): fix 1 flaky Mailpit test and share the stall guard Nov 2, 2026
@avallete
avallete enabled auto-merge November 2, 2026 16:51
@avallete
avallete added this pull request to the merge queue Nov 2, 2026
Merged via the queue into develop with commit 62cf836 Nov 2, 2026
54 checks passed
@avallete
avallete deleted the avallete/flaky-fixes-2026-10-24 branch November 2, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants