Skip to content

fix(cli): stop db reset failing at random (CLI-2312, CLI-2563) - #7380

Merged
7ttp merged 5 commits into
developfrom
7ttp/cli-2563-supabase-db-reset-intermittently-fails-temporary-realtime
Oct 29, 2026
Merged

7ttp merged 5 commits into
developfrom
7ttp/cli-2563-supabase-db-reset-intermittently-fails-temporary-realtime

Conversation

@7ttp

@7ttp 7ttp commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

TL;DR

stops supabase db reset randomly failing at Initialising schema...

whats hurting the user(s)?

supabase db reset fails at random with error running container: exit 1, mostly in CI

it recreated the db while realtime kept running, so docker restarted realtime mid reset and its migrations collided with the reset's own on schema_migrations_pkey

now fixed by:

storage, auth, realtime and the pooler are stopped before the db is removed and started once at the end,
even when the reset fails...

sequenceDiagram
    participant R as db reset
    participant S as storage, auth, realtime, pooler
    participant D as docker
    participant DB as db
    participant J as init jobs

    Note over R,J: before
    R->>DB: remove and recreate
    DB--xS: connection lost
    D->>S: restart (unless-stopped)
    par
        S->>DB: realtime migrations
    and
        J->>DB: realtime migrations
    end
    DB-->>J: schema_migrations_pkey
    J-->>R: exit 1, reset fails

    Note over R,J: now
    R->>S: stop
    R->>DB: remove and recreate
    J->>DB: init migrations
    R->>DB: your migrations + seed
    R->>S: start once, reload kong
    Note over R,S: same restart on failure or ctrl-c
Loading

ref:

@7ttp 7ttp self-assigned this Oct 28, 2026
@7ttp
7ttp requested a review from a team as a code owner October 28, 2026 19:20

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

Both independent reviews completed and reported no findings. Reading the changed code, surrounding setup and caller paths, tests, trusted conventions, and pending next-branch context revealed no concrete defects or fundamental conflicts. Tests were inspected but not rerun because dependencies are absent.

Findings

No issues found.

Stats

Claude findings: 0 · Codex findings: 0 · Confirmed: 0 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

@avallete
avallete self-requested a review October 29, 2026 11:40
@7ttp 7ttp changed the title fix(cli): stop db reset failing at random (CLI-2563) fix(cli): stop db reset failing at random (CLI-2312, CLI-2563) Oct 29, 2026

@avallete avallete left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dogfooded this against develop (097b7f2) on a real project (migrations + seed, pooler enabled, Docker runtime).

Performance looks good. Stopping the services before the reset adds about 1.5s per reset, and start, stop and the declarative shadow diff are unchanged:

Command PR median develop median
db reset (5 runs) 21.9s 20.3s
start (3 runs) 28.6s 28.6s
stop (3 runs) 3.5s 3.6s
db schema declarative sync (3 runs) 5.3s 5.2s

These also pass: a reset that fails on a bad migration (services come back healthy within 15s), db reset after db start (no extra services started), and db reset with the pooler and realtime disabled. All 5 PR resets were clean, with no schema_migrations_pkey error.

Blocking: a Ctrl-C or failure after the db container is removed leaves the stack broken.

In recreate-local-database.ts (around L325), the cleanup restarts storage, auth, realtime and the pooler even when the db container is missing or not running. docker restart succeeds, so no warning is printed, but the services crash-loop. Two ways to hit it:

1. Ctrl-C after the db is removed. 3 of 3 runs never recovered: the db container was gone, storage, auth and realtime were stuck in Restarting (1), and the API and auth returned 503 for over 90s.

supabase start
supabase db reset --debug   # press Ctrl-C as soon as "Recreating database" appears
docker ps -a --format '{{.Names}}\t{{.Status}}' | grep -E 'db|storage|auth|realtime|pooler'
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:54321/auth/v1/health   # 503

2. A failure while the db is being recreated. This isn't specific to Ctrl-C: any real failure at this step (image, volume, OOM) breaks the stack the same way, which is the CI scenario this PR targets. Killing the new db container just after it starts makes the reset exit 1 with HealthCheckTimeoutError. Storage, auth and realtime then crash-loop, and the API and auth return 503 with no recovery after 60s.

To repro, run a watcher that kills the new db container as soon as the reset recreates it, then run the reset in a second terminal:

supabase start
P=<project_id>   # from supabase/config.toml
OLD=$(docker inspect -f '{{.Id}}' supabase_db_$P)

# terminal 1: kill the new db container as soon as it appears, and keep docker from restarting it
while :; do
  ID=$(docker inspect -f '{{.Id}}' supabase_db_$P 2>/dev/null)
  if [ -n "$ID" ] && [ "$ID" != "$OLD" ]; then
    docker update --restart=no supabase_db_$P && docker kill supabase_db_$P && break
  fi
  sleep 0.1
done

# terminal 2
supabase db reset   # exits 1: HealthCheckTimeoutError "supabase_db_<id> container is not running: exited"
docker ps -a --format '{{.Names}}\t{{.Status}}' | grep -E 'db|storage|auth|realtime|pooler'
# db: Exited (137); storage/auth/realtime: Restarting (1)
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:54321/auth/v1/health   # 503

Recover with supabase stop --no-backup && supabase start.

Suggested fix: only restart the services once the db is usable again. Either check it in the cleanup and print a clear "run supabase stop && supabase start" message, or finish recreating the db before restarting them. Integration tests that interrupt right after the db is removed, and that fail the db health check during recreation, would cover both paths.

Nit: the comment above the cleanup (around L324) just describes the branches and can go.

@github-actions

Copy link
Copy Markdown
Contributor

CLI benchmark: no changes found

Commit 17855f8 against develop at 3a9e1fb · 10 runs each · report only, never blocks merging

Command Before After Change
supabase --version 180 ms 173 ms no real change
supabase --help 195 ms 183 ms no real change (-6%, within normal variation)
supabase init 302 ms 287 ms no real change
supabase migration new bench 267 ms 258 ms no real change
supabase gen types --lang typescript --project-id replayfixtureprojref 305 ms 291 ms no real change
supabase projects list 261 ms 276 ms no real change (+6%, within normal variation)
supabase functions list --project-ref replayfixtureprojref 258 ms 284 ms no real change (+10%, within normal variation)

API calls: unchanged for all 7 commands.

Binary size: 133.8 MB, up 4.0 KB from 133.8 MB.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command Before range After range Change Verdict
supabase --version 148-239 ms 148-217 ms -7 ms (-4%) within noise
supabase --help 158-235 ms 155-217 ms -12 ms (-6%) within noise
supabase init 204-359 ms 201-450 ms -15 ms (-5%) within noise
supabase migration new bench 213-407 ms 231-301 ms -9 ms (-3%) within noise
supabase gen types --lang typescript --project-id replayfixtureprojref 242-356 ms 266-328 ms -14 ms (-5%) within noise
supabase projects list 233-307 ms 244-295 ms +15 ms (+6%) within noise
supabase functions list --project-ref replayfixtureprojref 231-827 ms 254-628 ms +26 ms (+10%) within noise

API calls per command

Command Before After Endpoint changes
supabase --version 0 0
supabase --help 0 0
supabase init 0 0
supabase migration new bench 0 0
supabase gen types --lang typescript --project-id replayfixtureprojref 2 2
supabase projects list 1 1
supabase functions list --project-ref replayfixtureprojref 2 2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Comment /bench to run it again on the latest commit.

Workflow run

@7ttp
7ttp requested a review from avallete October 29, 2026 14:53

@avallete avallete left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-ran the two failing repros against 17855f8. Both are fixed:

  • Ctrl-C at "Recreating database" (3 runs): exits 130 and prints the new "local database is not running… left stopped" warning. Storage, auth, realtime and the pooler stay stopped instead of crash-looping.
  • New db container killed during recreate: exits 1 with HealthCheckTimeoutError and prints the same warning. The services stay stopped.

In every run, supabase stop && supabase start brought the stack back healthy (REST and auth return 200). The nit is addressed too. The normal reset path is unchanged, so the earlier benchmarks still hold.

LGTM 👍

@7ttp
7ttp enabled auto-merge October 29, 2026 19:31
@7ttp
7ttp added this pull request to the merge queue Oct 29, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 29, 2026
@7ttp
7ttp added this pull request to the merge queue Oct 29, 2026
Merged via the queue into develop with commit bcd3ec9 Oct 29, 2026
47 checks passed
@7ttp
7ttp deleted the 7ttp/cli-2563-supabase-db-reset-intermittently-fails-temporary-realtime branch October 29, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants