Skip to content

Commit fd2e017

Browse files
authored
fix(cli): recover start and db start from a stopped database container (#7369)
## Summary If a `supabase start` dies after Docker creates the database container but before it starts it, the container is left in Docker's `created` state. That can happen if the process is killed, a port is already taken, or a second `start` rolls back the first one's containers. Every `supabase start` after that printed `supabase start is already running.` and then failed with `container is not running: created`. The way out was to remove the container by hand, and nothing pointed you there. Looking into it turned up three more problems along the same path: - The leftover volume never gets set up. `start` decided whether to run the first-run setup (schema, roles, migrations, seed) purely on whether the database volume already existed. A leftover volume that Postgres never initialised still exists, so after `supabase stop` and `supabase start` the setup was skipped and the stack came up without `supabase_migrations` or any of the project's migrations. - Two starts on one project break each other. When one fails, its rollback prunes the project's containers and network by label, including the ones the other start is still bringing up. Usually both fail, and they can leave behind exactly the `created` container above. - `db start` only checked that the database container existed. For a stopped or never-started one it printed `Postgres database is already running.` and exited 0 with Postgres down. The local database auto-start in `db schema declarative generate` and `sync` used the same check, skipped the start and then failed to connect. This fixes all of them. ### What changes - `start` recovers a `created` database container the same way it already recovers an `exited` one. It removes the project's containers and network, keeps the volumes, and brings the stack up again. - A database container that isn't running and can't be recovered (an unknown state, or Bitbucket Pipelines) now fails with `<container> container is not running: <state>` and suggests `supabase stop`. It no longer prints `supabase start is already running.` first. - `start` and `db start` check whether a reused volume was ever initialised. Between `docker create` and `docker start` they run `docker cp <container>:/var/lib/postgresql/data/PG_VERSION -`, which reads through the volume mount while the container is still stopped. The Postgres image runs `initdb` when `PG_VERSION` is missing, so a missing file means the volume is uninitialised. In that case the first-run setup runs and bucket seeding happens as on a fresh volume, and a failed bring-up prunes the volume like it does for a fresh one. Only Docker's or Podman's own message for a missing `PG_VERSION` counts as uninitialised. Any other outcome, including a connection error, is treated as initialised, so an ambiguous probe never runs setup over real data. - `start` and `db start` take a per-project lock at `$SUPABASE_HOME/locks/start-<project_id>.lock`. It's held through bring-up and rollback. A second start on the same project prints `Waiting for another supabase start for this project to finish...` and waits for the first, up to 10 minutes, before failing with `LocalStackStartBusyError`. If the lock file can't be created, the start carries on without it rather than failing. `stop` doesn't take the lock. - `db start` and the declarative auto-start now read the container's state as well as whether it exists. A database container that isn't running is removed on its own with `docker container rm -f`, keeping the volume, and brought up again. The `PG_VERSION` check above decides whether setup runs. The project's other containers are left alone, since `db start` only manages the database. In Bitbucket Pipelines they fail with the same not-running error as `start` instead. `db reset` and `db diff` still only check that the container exists, since `db reset` recreates it anyway. - If `db start` fails, it now rolls back only what it creates: the database container, the database volume when this run created it or found it uninitialised, and any project network nothing else is using. It used to remove every container in the project and, on a fresh volume, prune every unused project volume, so a failed recovery could take down the rest of a running stack. A stopped database with `--from-backup` is refused before anything is removed. - Under `--debug`, a missing profile file now logs `No profile file at <path>; using the default profile.` instead of the raw file-not-found error. That line was the only thing in the original report that looked like an error, and it wasn't one. ### Not covered - If `start` is killed after Postgres has started running `initdb` but before the CLI's setup finishes, the volume has `PG_VERSION`, so the next start treats it as initialised and skips setup. That's the same as before this change. Catching it would need a marker written once setup completes. I'd rather not guess at that from schema contents, since a project with no migrations looks the same as an unfinished one. ## Linked issue Closes #7344 (CLI-2876)
1 parent 96781a4 commit fd2e017

31 files changed

Lines changed: 2063 additions & 313 deletions

‎apps/cli/src/command-internal/db-bootstrap/container-lifecycle.ts‎

Lines changed: 41 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
* Given a fully-resolved {@link StartContainerSpec}, sets the two project-identity labels,
33
* provisions the container's own named volumes, builds the `docker create` argv, spawns `docker
44
* create`, copies any `secretFiles` in via `docker cp` (see
5-
* {@link copyStartSecretFilesIntoContainer}), then spawns `docker start`.
5+
* {@link copyStartSecretFilesIntoContainer}), runs the caller's optional `beforeStart` hook, then
6+
* spawns `docker start`.
67
*
78
* Network creation is not part of this per-container function; see {@link ensureNetwork} for why
89
* it runs once instead of once per container.
@@ -36,6 +37,7 @@ import {
3637
isDockerClientEnvKey,
3738
type StartContainerSpec,
3839
} from "./docker-create-args.ts";
40+
import { POSTGRES_DATA_DIR } from "./postgres.service.ts";
3941

4042
type StartSecretFileSpec = NonNullable<StartContainerSpec["secretFiles"]>[number];
4143

@@ -334,7 +336,7 @@ export class VolumeInspectError extends Data.TaggedError("VolumeInspectError")<{
334336
}
335337

336338
/** Docker's/Podman's "no such volume" stderr shape for `volume inspect`. */
337-
function isVolumeNotFoundMessage(message: string): boolean {
339+
export function isVolumeNotFoundMessage(message: string): boolean {
338340
return /no such volume/iu.test(message);
339341
}
340342

@@ -645,17 +647,50 @@ function extractPreStartArchiveIntoContainer(
645647
);
646648
}
647649

650+
/** Docker's, Podman's and older Docker clients' `cp` diagnostics for a missing `PG_VERSION`. */
651+
const PG_VERSION_MISSING_PATTERN =
652+
/could not find the file \S*\/PG_VERSION in container|"\S*\/PG_VERSION" could not be found on container|no such container:path: \S+:\S*\/PG_VERSION/iu;
653+
654+
/**
655+
* Whether the Postgres data directory of a created, not-yet-started container already holds a
656+
* cluster (`PG_VERSION`, written by `initdb`). Only a diagnostic reporting `PG_VERSION` itself
657+
* as missing resolves `false`; every other outcome resolves `true`, so an ambiguous probe never
658+
* runs setup over real data.
659+
*/
660+
export const postgresDataInitialized = Effect.fn("ContainerLifecycle.postgresDataInitialized")(
661+
function* (spawner: Spawner, containerId: string) {
662+
const initialized = yield* Effect.scoped(
663+
Effect.gen(function* () {
664+
const child = yield* spawnContainerCli(
665+
spawner,
666+
["cp", `${containerId}:${POSTGRES_DATA_DIR}/PG_VERSION`, "-"],
667+
{ stdin: "ignore", stdout: "ignore", stderr: "pipe" },
668+
);
669+
const [exitCode, stderr] = yield* Effect.all(
670+
[child.exitCode.pipe(Effect.map(Number)), collectText(child.stderr)],
671+
{ concurrency: "unbounded" },
672+
);
673+
return exitCode === 0 || !PG_VERSION_MISSING_PATTERN.test(stderr);
674+
}),
675+
).pipe(Effect.orElseSucceed(() => true));
676+
yield* Effect.annotateCurrentSpan({ "db.data_initialized": initialized });
677+
return initialized;
678+
},
679+
);
680+
648681
/**
649682
* Provisions this container's named volumes (skipped under Bitbucket Pipelines), creates it,
650683
* copies any `secretFiles` and `preStartArchives` in via `docker cp` while it's created but not
651-
* yet running (so its entrypoint never races the copy), then starts it.
684+
* yet running (so its entrypoint never races the copy), runs `beforeStart` with the container id
685+
* while it is still stopped, then starts it.
652686
*
653687
* Resolves to the created container's id/name on success.
654688
*/
655689
export function createContainer(
656690
spawner: Spawner,
657691
spec: StartContainerSpec,
658692
opts: ContainerOpts,
693+
beforeStart?: (containerId: string) => Effect.Effect<void>,
659694
): Effect.Effect<string, ContainerError> {
660695
return Effect.gen(function* () {
661696
const labels: Record<string, string> = {
@@ -711,6 +746,9 @@ export function createContainer(
711746
}).pipe(Effect.orElseSucceed(() => 0)),
712747
),
713748
);
749+
if (beforeStart !== undefined) {
750+
yield* beforeStart(containerId);
751+
}
714752
yield* dockerStartContainer(spawner, containerId, finalSpec);
715753
return containerId;
716754
}).pipe(

‎apps/cli/src/command-internal/db-bootstrap/container-lifecycle.unit.test.ts‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import {
1515
removeContainer,
1616
removeVolume,
1717
createContainer,
18+
postgresDataInitialized,
1819
volumeExists,
1920
} from "./container-lifecycle.ts";
2021
import { useTempWorkdir } from "../../../tests/helpers/command-mocks.ts";
@@ -524,6 +525,40 @@ describe("createContainer secretFiles", () => {
524525
});
525526
});
526527

528+
describe("createContainer beforeStart", () => {
529+
it.live(
530+
"runs the hook with the created container id after the secret copy and before start",
531+
() => {
532+
const mock = mockSpawner((args) => {
533+
if (args[0] === "create") return { exitCode: 0, stdout: "container-id-hook\n" };
534+
return { exitCode: 0 };
535+
});
536+
const spec: StartContainerSpec = {
537+
...baseSpec,
538+
binds: [],
539+
secretFiles: [{ containerPath: "/etc/kong/kong.yml", content: "super-secret-content" }],
540+
};
541+
542+
return createContainer(
543+
mock.spawner,
544+
spec,
545+
{ projectId: "proj", isBitbucketPipeline: false, workdir: workdir.current, extraHosts: [] },
546+
(containerId) => postgresDataInitialized(mock.spawner, containerId).pipe(Effect.asVoid),
547+
).pipe(
548+
Effect.map((containerId) => {
549+
expect(containerId).toBe("container-id-hook");
550+
expect(mock.spawned).toEqual([
551+
expect.arrayContaining(["create"]),
552+
["cp", "-", "container-id-hook:/"],
553+
["cp", "container-id-hook:/var/lib/postgresql/data/PG_VERSION", "-"],
554+
["start", "container-id-hook"],
555+
]);
556+
}),
557+
);
558+
},
559+
);
560+
});
561+
527562
describe("ensureNetwork", () => {
528563
it.live("creates the network with labels when it does not exist yet", () => {
529564
const mock = mockSpawner((args) =>
@@ -728,6 +763,88 @@ describe("volumeExists", () => {
728763
});
729764
});
730765

766+
describe("postgresDataInitialized", () => {
767+
const pgVersionProbe = ["cp", "container-id-pg:/var/lib/postgresql/data/PG_VERSION", "-"];
768+
769+
it.live("resolves true when PG_VERSION can be copied out (cp exits 0)", () => {
770+
const mock = mockSpawner(() => ({ exitCode: 0 }));
771+
return postgresDataInitialized(mock.spawner, "container-id-pg").pipe(
772+
Effect.map((initialized) => {
773+
expect(initialized).toBe(true);
774+
expect(mock.spawned).toEqual([pgVersionProbe]);
775+
}),
776+
);
777+
});
778+
779+
const missingFileStderr = {
780+
Docker:
781+
"Error response from daemon: Could not find the file /var/lib/postgresql/data/PG_VERSION in container container-id-pg\n",
782+
Podman:
783+
'Error: "/var/lib/postgresql/data/PG_VERSION" could not be found on container container-id-pg: no such file or directory\n',
784+
"an older Docker client":
785+
"Error: No such container:path: container-id-pg:/var/lib/postgresql/data/PG_VERSION\n",
786+
"Docker after a config warning":
787+
"WARNING: Error loading config file: open /home/me/.docker/config.json: no such file or directory\nError response from daemon: Could not find the file /var/lib/postgresql/data/PG_VERSION in container container-id-pg\n",
788+
};
789+
for (const [runtime, stderr] of Object.entries(missingFileStderr)) {
790+
it.live(`resolves false when ${runtime} reports the PG_VERSION file missing`, () => {
791+
const mock = mockSpawner(() => ({ exitCode: 1, stderr }));
792+
return postgresDataInitialized(mock.spawner, "container-id-pg").pipe(
793+
Effect.map((initialized) => {
794+
expect(initialized).toBe(false);
795+
}),
796+
);
797+
});
798+
}
799+
800+
const otherFailureStderr = {
801+
"a missing container": "Error response from daemon: No such container: container-id-pg\n",
802+
"a permission error": "Error response from daemon: permission denied\n",
803+
"an unreachable daemon": "Cannot connect to the Docker daemon at unix:///var/run/docker.sock\n",
804+
"an unreadable Docker CLI config":
805+
"open /home/me/.docker/config.json: no such file or directory\n",
806+
"a Docker socket that does not exist":
807+
'error during connect: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: no such file or directory\n',
808+
"a Docker socket that does not exist when requesting the PG_VERSION archive":
809+
'error during connect: Head "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/container-id-pg/archive?path=%2Fvar%2Flib%2Fpostgresql%2Fdata%2FPG_VERSION": dial unix /var/run/docker.sock: connect: no such file or directory\n',
810+
"a Podman socket that does not exist":
811+
'Error: unable to connect to Podman socket: Get "http://d/v5.2.0/libpod/_ping": dial unix /run/user/1000/podman/podman.sock: connect: no such file or directory\n',
812+
"a Podman socket that does not exist when requesting the PG_VERSION archive":
813+
'Error: unable to connect to Podman socket: Get "http://d/v5.2.0/libpod/containers/container-id-pg/archive?path=%2Fvar%2Flib%2Fpostgresql%2Fdata%2FPG_VERSION": dial unix /run/user/1000/podman/podman.sock: connect: no such file or directory\n',
814+
"a Docker config warning beside an unrelated PG_VERSION error":
815+
"WARNING: Error loading config file: open /home/me/.docker/config.json: no such file or directory\nError response from daemon: permission denied reading /var/lib/postgresql/data/PG_VERSION\n",
816+
"empty stderr": "",
817+
};
818+
for (const [cause, stderr] of Object.entries(otherFailureStderr)) {
819+
it.live(`resolves true (never re-seeds) on a non-zero exit with ${cause}`, () => {
820+
const mock = mockSpawner(() => ({ exitCode: 1, stderr }));
821+
return postgresDataInitialized(mock.spawner, "container-id-pg").pipe(
822+
Effect.map((initialized) => {
823+
expect(initialized).toBe(true);
824+
}),
825+
);
826+
});
827+
}
828+
829+
it.live("resolves true when no container runtime can be spawned", () => {
830+
const spawner = ChildProcessSpawner.make(() =>
831+
Effect.fail(
832+
PlatformError.systemError({
833+
_tag: "NotFound",
834+
module: "ChildProcess",
835+
method: "spawn",
836+
description: "spawn ENOENT",
837+
}),
838+
),
839+
);
840+
return postgresDataInitialized(spawner, "container-id-pg").pipe(
841+
Effect.map((initialized) => {
842+
expect(initialized).toBe(true);
843+
}),
844+
);
845+
});
846+
});
847+
731848
describe("removeContainer", () => {
732849
it.live("spawns `docker container rm -f <id>` and succeeds on exit 0", () => {
733850
const mock = mockSpawner(() => ({ exitCode: 0 }));

0 commit comments

Comments
 (0)