Repository navigation
Commit fd2e017
authored
fix(cli): recover start and db start from a stopped database container (#7369)
## Summary
If a `supabase start` dies after Docker creates the database container
but before it starts it, the container is left in Docker's `created`
state. That can happen if the process is killed, a port is already
taken, or a second `start` rolls back the first one's containers. Every
`supabase start` after that printed `supabase start is already running.`
and then failed with `container is not running: created`. The way out
was to remove the container by hand, and nothing pointed you there.
Looking into it turned up three more problems along the same path:
- The leftover volume never gets set up. `start` decided whether to run
the first-run setup (schema, roles, migrations, seed) purely on whether
the database volume already existed. A leftover volume that Postgres
never initialised still exists, so after `supabase stop` and `supabase
start` the setup was skipped and the stack came up without
`supabase_migrations` or any of the project's migrations.
- Two starts on one project break each other. When one fails, its
rollback prunes the project's containers and network by label, including
the ones the other start is still bringing up. Usually both fail, and
they can leave behind exactly the `created` container above.
- `db start` only checked that the database container existed. For a
stopped or never-started one it printed `Postgres database is already
running.` and exited 0 with Postgres down. The local database auto-start
in `db schema declarative generate` and `sync` used the same check,
skipped the start and then failed to connect.
This fixes all of them.
### What changes
- `start` recovers a `created` database container the same way it
already recovers an `exited` one. It removes the project's containers
and network, keeps the volumes, and brings the stack up again.
- A database container that isn't running and can't be recovered (an
unknown state, or Bitbucket Pipelines) now fails with `<container>
container is not running: <state>` and suggests `supabase stop`. It no
longer prints `supabase start is already running.` first.
- `start` and `db start` check whether a reused volume was ever
initialised. Between `docker create` and `docker start` they run `docker
cp <container>:/var/lib/postgresql/data/PG_VERSION -`, which reads
through the volume mount while the container is still stopped. The
Postgres image runs `initdb` when `PG_VERSION` is missing, so a missing
file means the volume is uninitialised. In that case the first-run setup
runs and bucket seeding happens as on a fresh volume, and a failed
bring-up prunes the volume like it does for a fresh one. Only Docker's
or Podman's own message for a missing `PG_VERSION` counts as
uninitialised. Any other outcome, including a connection error, is
treated as initialised, so an ambiguous probe never runs setup over real
data.
- `start` and `db start` take a per-project lock at
`$SUPABASE_HOME/locks/start-<project_id>.lock`. It's held through
bring-up and rollback. A second start on the same project prints
`Waiting for another supabase start for this project to finish...` and
waits for the first, up to 10 minutes, before failing with
`LocalStackStartBusyError`. If the lock file can't be created, the start
carries on without it rather than failing. `stop` doesn't take the lock.
- `db start` and the declarative auto-start now read the container's
state as well as whether it exists. A database container that isn't
running is removed on its own with `docker container rm -f`, keeping the
volume, and brought up again. The `PG_VERSION` check above decides
whether setup runs. The project's other containers are left alone, since
`db start` only manages the database. In Bitbucket Pipelines they fail
with the same not-running error as `start` instead. `db reset` and `db
diff` still only check that the container exists, since `db reset`
recreates it anyway.
- If `db start` fails, it now rolls back only what it creates: the
database container, the database volume when this run created it or
found it uninitialised, and any project network nothing else is using.
It used to remove every container in the project and, on a fresh volume,
prune every unused project volume, so a failed recovery could take down
the rest of a running stack. A stopped database with `--from-backup` is
refused before anything is removed.
- Under `--debug`, a missing profile file now logs `No profile file at
<path>; using the default profile.` instead of the raw file-not-found
error. That line was the only thing in the original report that looked
like an error, and it wasn't one.
### Not covered
- If `start` is killed after Postgres has started running `initdb` but
before the CLI's setup finishes, the volume has `PG_VERSION`, so the
next start treats it as initialised and skips setup. That's the same as
before this change. Catching it would need a marker written once setup
completes. I'd rather not guess at that from schema contents, since a
project with no migrations looks the same as an unfinished one.
## Linked issue
Closes #7344 (CLI-2876)1 parent 96781a4 commit fd2e017
31 files changed
Lines changed: 2063 additions & 313 deletions
File tree
- apps/cli
- src
- command-internal
- db-bootstrap
- commands
- db
- reset
- schema/declarative
- generate
- sync
- start
- start
- config
- tests/helpers
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
| 40 | + | |
39 | 41 | | |
40 | 42 | | |
41 | 43 | | |
| |||
334 | 336 | | |
335 | 337 | | |
336 | 338 | | |
337 | | - | |
| 339 | + | |
338 | 340 | | |
339 | 341 | | |
340 | 342 | | |
| |||
645 | 647 | | |
646 | 648 | | |
647 | 649 | | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
648 | 681 | | |
649 | 682 | | |
650 | 683 | | |
651 | | - | |
| 684 | + | |
| 685 | + | |
652 | 686 | | |
653 | 687 | | |
654 | 688 | | |
655 | 689 | | |
656 | 690 | | |
657 | 691 | | |
658 | 692 | | |
| 693 | + | |
659 | 694 | | |
660 | 695 | | |
661 | 696 | | |
| |||
711 | 746 | | |
712 | 747 | | |
713 | 748 | | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
714 | 752 | | |
715 | 753 | | |
716 | 754 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
| |||
524 | 525 | | |
525 | 526 | | |
526 | 527 | | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
527 | 562 | | |
528 | 563 | | |
529 | 564 | | |
| |||
728 | 763 | | |
729 | 764 | | |
730 | 765 | | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
731 | 848 | | |
732 | 849 | | |
733 | 850 | | |
| |||
0 commit comments