Repository navigation
fix(cli): start the local stack when Docker cannot exec health checks - #7362
Merged
Coly010 merged 2 commits intoOct 28, 2026
Conversation
On nested or sandboxed Docker hosts every HEALTHCHECK fails to exec (ExitCode -1, "OCI runtime exec failed"), so container health never reaches healthy and `supabase start` / `db start` time out and roll back. - Gate Postgres readiness on a host-side connect probe for start, db start, db reset and the db diff / migration squash shadows, then wait up to 15s for Docker's first health verdict so `status` keeps working. - When a container's health check cannot exec, fall back to probing its Kong route (or any Kong response for Kong itself), else accept it as running, and print one warning naming the unverified containers. - Accept an exec-failed db in `status` and the already-running gate. - Include the last health-check output in not-ready errors. - Bound gateway probes with one timeout and cap the service wait in wall-clock time. Closes #7325
Contributor
There was a problem hiding this comment.
🤖 AI Review
Both independent reviews were available. All four distinct findings were verified: three minor correctness or reporting issues, plus an advisory nit about the unchanged reset storage gate. No findings overlap.
Findings
| Severity | Location | Category | Sources | Claim |
|---|---|---|---|---|
| 🟡 MINOR | apps/cli/src/command-internal/db-bootstrap/health-check.ts:417 |
correctness |
claude | The wall-clock cap can terminate the wait before its configured retry count is exhausted because HTTP probes run sequentially, while the cap allows only one HTTP timeout per round. |
| 🟡 MINOR | apps/cli/src/command-internal/docker-lifecycle.ts:268 |
correctness |
codex | Legacy runc image faults can be classified as host exec failures, allowing a broken health-check command to pass through the running fallback. |
| 🟡 MINOR | apps/cli/src/commands/start/start.handler.ts:1891 |
error-handling |
codex | Startup with --ignore-health-check can succeed without identifying services whose readiness was assumed through the running fallback. |
| ⚪ NIT | apps/cli/src/command-internal/db-bootstrap/await-storage-ready.ts:40 |
consistency |
claude | The reset storage readiness gate does not supply an exec-failure fallback. If reached with a running Storage container whose Docker health check cannot execute, it still exhausts the readiness wait. |
Findings outside the diff
- ⚪ NIT
apps/cli/src/command-internal/db-bootstrap/await-storage-ready.ts:40— The reset storage readiness gate does not supply an exec-failure fallback. If reached with a running Storage container whose Docker health check cannot execute, it still exhausts the readiness wait.
Stats
Claude findings: 2 · Codex findings: 2 · Confirmed: 4 · Refuted: 0 · Uncertain: 0
Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run
This review runs once per PR. A maintainer can request another with a /ai-review comment.
…ing under --ignore-health-check - Treat pre-1.1 runc "exec user process caused: ..." exec format, missing-interpreter and permission errors as image faults, not as a host that cannot exec. - Carry the containers accepted on `running` alone on HealthCheckTimeoutError and print the unverified-readiness warning when --ignore-health-check downgrades a failure.
jgoux
approved these changes
Oct 28, 2026
Coly010
deleted the
columferry/cli-2855-local-supabase-startup-fails-with-healthchecktimeouterror-in
branch
October 28, 2026 10:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On some nested or sandboxed Docker hosts,
docker startworks butdocker execdoesn't. runc can't join the container's namespaces, so every exec fails withOCI runtime exec failed: … error executing setns process: exit status 1. Docker runs everyHEALTHCHECKthrough exec, so on these hosts no container ever turns healthy.supabase startwaited on Docker health for Postgres, so it sat out the wholedb.health_timeout(2 minutes by default), failed withHealthCheckTimeoutError … container is not ready: unhealthyand rolled the stack back. The real reason, the setns error inState.Health.Log, was never shown.This makes the readiness checks work without Docker being able to exec into containers, and puts the health-check output in the error when something genuinely isn't ready.
What changes
Postgres readiness is a connection from the host.
start,db start,db reset, and the shadow databases fordb diffandmigration squashnow wait until Postgres accepts a connection on its published port. It's the same probe the pg-delta shadow already used. The db gate doesn't read Docker health any more.Other services fall back when their health check can't run. That means the last health check failed to exec (exit code -1,
OCI runtime exec failed), and the failure isn't a missing or broken health-check command in the image. In that case the gate checks:startthen prints one warning that names the containers it could only confirm as running.statusand the already-running check onstartaccept a db whose health check can't exec, with the same warning, instead of failing.Not-ready errors say why. They now include the last health-check exit code and output, e.g.
container is not ready: unhealthy (last health check exit code -1: OCI runtime exec failed: …). When exec is the cause, the error adds a hint with adocker exec <container> truecommand to confirm it.Probes are bounded. One timeout covers each HTTP probe's request and body, and the service wait has a wall-clock cap. A proxy that accepts connections and then hangs can't stretch the wait out indefinitely.
On a host where exec works, nothing falls back and no warning is printed.
Trade-offs of always probing Postgres from the host
I went with always using the connection probe for Postgres, rather than only falling back to it when exec fails. It's simpler, and a working connection is what we actually care about. It does change a few things on normal hosts too:
startis faster. Postgres accepts connections several seconds before Docker's first health check runs (its interval is 10s). In my runsstartreturned about 9s sooner, roughly a third faster.db startgives most of that back.statusand the already-running check still read Docker health, sodb startanddb resetwait up to 15s afterwards for Docker's first health verdict. Without that wait,supabase db start && supabase statuswould fail withcontainer is not ready: starting. Sodb startends up only slightly faster than before.startdoes the same wait, but by the time the other services are ready the db has almost always settled, so it costs nothing on a full stack.db.health_timeoutruns out (2 minutes by default). Before, thepg_isreadyhealth check didn't authenticate, so the gate passed and the next step failed straight away. I don't think that case is common enough to special-case, but I'm happy to make an auth failure end the wait early if we'd rather.db start --from-backupstill ignores a readiness timeout, but the wait is now a connection from the host to the published port rather than Docker health.db diffstarts for pg-delta (throughstartLocalDatabase) gets the same 15s settle, so its timing stays about the same as before.Not covered
db reseton these hosts still fails at the Kong reload step, becausekong reloadruns throughdocker exec. It now fails in a few seconds withKongReloadErrorinstead of after the 2-minute timeout.docker psstill shows these containers asunhealthy. That's Docker's own state and the CLI can't change it.Linked issue
Closes #7325 (CLI-2855)