Skip to content

fix(cli): start the local stack when Docker cannot exec health checks - #7362

Merged
Coly010 merged 2 commits into
developfrom
columferry/cli-2855-local-supabase-startup-fails-with-healthchecktimeouterror-in
Oct 28, 2026
Merged

Coly010 merged 2 commits into
developfrom
columferry/cli-2855-local-supabase-startup-fails-with-healthchecktimeouterror-in

Conversation

@Coly010

@Coly010 Coly010 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Summary

On some nested or sandboxed Docker hosts, docker start works but docker exec doesn't. runc can't join the container's namespaces, so every exec fails with OCI runtime exec failed: … error executing setns process: exit status 1. Docker runs every HEALTHCHECK through exec, so on these hosts no container ever turns healthy.

supabase start waited on Docker health for Postgres, so it sat out the whole db.health_timeout (2 minutes by default), failed with HealthCheckTimeoutError … container is not ready: unhealthy and rolled the stack back. The real reason, the setns error in State.Health.Log, was never shown.

This makes the readiness checks work without Docker being able to exec into containers, and puts the health-check output in the error when something genuinely isn't ready.

What changes

  • Postgres readiness is a connection from the host. start, db start, db reset, and the shadow databases for db diff and migration squash now wait until Postgres accepts a connection on its published port. It's the same probe the pg-delta shadow already used. The db gate doesn't read Docker health any more.

  • Other services fall back when their health check can't run. That means the last health check failed to exec (exit code -1, OCI runtime exec failed), and the failure isn't a missing or broken health-check command in the image. In that case the gate checks:

    • the Kong-routed services (Auth, Storage, postgres-meta, Analytics) through their health route in Kong, which has to return 200
    • Kong itself by any HTTP response from its proxy
    • everything else by it running

    start then prints one warning that names the containers it could only confirm as running.

  • status and the already-running check on start accept a db whose health check can't exec, with the same warning, instead of failing.

  • Not-ready errors say why. They now include the last health-check exit code and output, e.g. container is not ready: unhealthy (last health check exit code -1: OCI runtime exec failed: …). When exec is the cause, the error adds a hint with a docker exec <container> true command to confirm it.

  • Probes are bounded. One timeout covers each HTTP probe's request and body, and the service wait has a wall-clock cap. A proxy that accepts connections and then hangs can't stretch the wait out indefinitely.

On a host where exec works, nothing falls back and no warning is printed.

Trade-offs of always probing Postgres from the host

I went with always using the connection probe for Postgres, rather than only falling back to it when exec fails. It's simpler, and a working connection is what we actually care about. It does change a few things on normal hosts too:

  • start is faster. Postgres accepts connections several seconds before Docker's first health check runs (its interval is 10s). In my runs start returned about 9s sooner, roughly a third faster.
  • db start gives most of that back. status and the already-running check still read Docker health, so db start and db reset wait up to 15s afterwards for Docker's first health verdict. Without that wait, supabase db start && supabase status would fail with container is not ready: starting. So db start ends up only slightly faster than before. start does the same wait, but by the time the other services are ready the db has almost always settled, so it costs nothing on a full stack.
  • A changed postgres password no longer fails fast. Say a reused volume has a different password from the one the CLI connects with. The connection probe keeps getting rejected until db.health_timeout runs out (2 minutes by default). Before, the pg_isready health check didn't authenticate, so the gate passed and the next step failed straight away. I don't think that case is common enough to special-case, but I'm happy to make an auth failure end the wait early if we'd rather.
  • db start --from-backup still ignores a readiness timeout, but the wait is now a connection from the host to the published port rather than Docker health.
  • The db that db diff starts for pg-delta (through startLocalDatabase) gets the same 15s settle, so its timing stays about the same as before.

Not covered

  • db reset on these hosts still fails at the Kong reload step, because kong reload runs through docker exec. It now fails in a few seconds with KongReloadError instead of after the 2-minute timeout.
  • docker ps still shows these containers as unhealthy. That's Docker's own state and the CLI can't change it.

Linked issue

Closes #7325 (CLI-2855)

On nested or sandboxed Docker hosts every HEALTHCHECK fails to exec
(ExitCode -1, "OCI runtime exec failed"), so container health never
reaches healthy and `supabase start` / `db start` time out and roll back.

- Gate Postgres readiness on a host-side connect probe for start, db start,
  db reset and the db diff / migration squash shadows, then wait up to 15s
  for Docker's first health verdict so `status` keeps working.
- When a container's health check cannot exec, fall back to probing its
  Kong route (or any Kong response for Kong itself), else accept it as
  running, and print one warning naming the unverified containers.
- Accept an exec-failed db in `status` and the already-running gate.
- Include the last health-check output in not-ready errors.
- Bound gateway probes with one timeout and cap the service wait in
  wall-clock time.

Closes #7325
@Coly010 Coly010 self-assigned this Oct 28, 2026
@Coly010
Coly010 requested a review from a team as a code owner October 28, 2026 09:40

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

Both independent reviews were available. All four distinct findings were verified: three minor correctness or reporting issues, plus an advisory nit about the unchanged reset storage gate. No findings overlap.

Findings

Severity Location Category Sources Claim
🟡 MINOR apps/cli/src/command-internal/db-bootstrap/health-check.ts:417 correctness claude The wall-clock cap can terminate the wait before its configured retry count is exhausted because HTTP probes run sequentially, while the cap allows only one HTTP timeout per round.
🟡 MINOR apps/cli/src/command-internal/docker-lifecycle.ts:268 correctness codex Legacy runc image faults can be classified as host exec failures, allowing a broken health-check command to pass through the running fallback.
🟡 MINOR apps/cli/src/commands/start/start.handler.ts:1891 error-handling codex Startup with --ignore-health-check can succeed without identifying services whose readiness was assumed through the running fallback.
⚪ NIT apps/cli/src/command-internal/db-bootstrap/await-storage-ready.ts:40 consistency claude The reset storage readiness gate does not supply an exec-failure fallback. If reached with a running Storage container whose Docker health check cannot execute, it still exhausts the readiness wait.

Findings outside the diff

  • ⚪ NIT apps/cli/src/command-internal/db-bootstrap/await-storage-ready.ts:40 — The reset storage readiness gate does not supply an exec-failure fallback. If reached with a running Storage container whose Docker health check cannot execute, it still exhausts the readiness wait.

Stats

Claude findings: 2 · Codex findings: 2 · Confirmed: 4 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

…ing under --ignore-health-check

- Treat pre-1.1 runc "exec user process caused: ..." exec format,
  missing-interpreter and permission errors as image faults, not as a
  host that cannot exec.
- Carry the containers accepted on `running` alone on
  HealthCheckTimeoutError and print the unverified-readiness warning when
  --ignore-health-check downgrades a failure.
@Coly010
Coly010 added this pull request to the merge queue Oct 28, 2026
Merged via the queue into develop with commit dbc4dfb Oct 28, 2026
59 checks passed
@Coly010
Coly010 deleted the columferry/cli-2855-local-supabase-startup-fails-with-healthchecktimeouterror-in branch October 28, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local Supabase startup fails with HealthCheckTimeoutError in Replit

2 participants