Merge branch '755-clone-major-upgrade' into 'master'

feat: clone major upgrade

Closes #756

See merge request postgres-ai/database-lab!1180

Author: Sarumyan

CLAUDE.md

@@ -13,6 +13,10 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - **Config projection** (`engine/pkg/models/configuration.go` + `engine/pkg/util/projection/`) walks only top-level struct fields with single flat `proj:"..."` paths — no recursion into nested structs. The `RetrievalMode` field on `ConfigProjection` carries no `proj:` tag because it's a synthetic field injected by `projectedAdminConfig` after `StoreJSON`; the request-side dispatcher reads it directly from the incoming JSON map. A projection field for a key that is **not pre-seeded** in the config scaffold (e.g. `connectionString`) needs the `,createKey` tag — `projection.Set` silently skips a missing leaf without it.
 - **Mode-aware config writes**: `applyProjectedAdminConfig` dispatches on the synthetic `retrievalMode`, and `guardModeFields` enforces a per-mode allow-list of populated fields so logical-only fields cannot leak into a physical config (and vice-versa).
 - **Source connection-string passthrough**: when `logicalDump.options.source.connectionString` is set it wins over the discrete `connection.*` fields. `logical/connstring.go` (`withDatabase`, `sourcePgxConfig`) preserves every libpq option (sslmode, connect_timeout, …) end-to-end into `pg_dump` (`-d <conninfo>`) and the engine's own pgx connections to the **source** (`getDBList`, `dbSourceActivity`). The password is always injected separately — never embedded in the string. Restore paths target the local container and must never receive it.
+- **Clone major upgrade** (`engine/internal/{provision,cloning,srv}/upgrade.go` + `scripts/pg_upgrade_clone.sh` + `Dockerfile.pg-upgrade`): `pg_upgrade --link` converts a clone's data directory in place. Three invariants hold the design together:
+  - **`FATAL` is a data-destroying status.** `filterRunningClones` (`cloning/storage.go`) drops any wrapper whose status is `FATAL` *or* whose container is not running, and `cleanupInvalidClones` then destroys its dataset. So a running clone must never be marked `FATAL`; failures that leave the clone up use `StatusWarning`. Anything that has to survive a restart must also be persisted with `SaveClonesState()` — `UpdateCloneStatus` only mutates memory.
+  - **Recovery is decided from the data directory, not from a breadcrumb — and only a positive marker proves completion.** Every passive marker in the directory is already in place *before* the transfer begins: a target-major `PG_VERSION` in `data_new` is written by the script's `initdb`, and `pg_upgrade` renames `global/pg_control` away when linking **starts**, not when it finishes (see "Reverting to old cluster" in the `pg_upgrade` docs). So a run killed part-way through the transfer is indistinguishable from a finished one by those facts alone. The script therefore writes `data_new/.dblab_upgrade_done` once `pg_upgrade` has returned 0, and `classifyOutcome` promotes only on a completed swap or on that marker (`ConversionDone`); everything else with a disabled old cluster is a `recoveryReprovision`. Treating the `pg_control` rename as proof of completion would promote a partially linked cluster, report success, and then delete the old directory holding the only copy of whatever had not been linked yet. The marker is removed once the swap is promoted, so it can never vouch for a later upgrade. Exit codes are carried into the status message by `describeExitCode` and never classify anything. `Base.RecoverInterruptedUpgrades` runs from `Run` *before* `filterRunningClones` for the same reason as the first invariant, and a missing `upgrade/state.json` means there is nothing in flight (`UpgradeUnchanged`), never a failure.
+  - **The engine owns every directory operation inside the clone directory.** It belongs to the engine OS user while the upgrade container runs as the data-directory owner (resolved by stat'ing it, not assumed — the image entrypoint gosu-chowns PGDATA to uid 999 while `zfs.go` chowns the dataset to `user.Current()`). The container therefore cannot create `data_new` or rename anything; the engine pre-creates the empty target, chowns it plus the `upgrade/` working area, and performs the swap itself.
 - **Engine-side image resolution**: `probe/registry.go` resolves a glibc-aware docker image by querying the live registries (Docker Hub for the generic image, GitLab for managed-provider SE images) with a per-repo TTL cache and a `go:embed` offline snapshot (`images_fallback.json`). Resolution never hard-fails or hangs: fresh cache → live fetch (bounded) → last good cache → embedded snapshot → provider default `<repo>:<major>`. Tag selection (`imageselect.go`) is pure and unit-tested; the `Registry` is built once on `Server`.
 
 ## Testing Before Pushing

README.md

@@ -120,12 +120,95 @@ Read more:
     - "Deletion protection" for clones, branches, and snapshots (blocks manual and auto-deletion, plus count-based retention for snapshots)
     - Persistent clones withstand DBLab restarts
     - "Reset" command for data version switching
+    - "Upgrade" command to move a single clone to a newer PostgreSQL major version in place
     - Resource quotas: CPU, RAM
 - Monitoring & security
     - `/healthz` API endpoint (no auth), extended `/status` endpoint ([API docs](https://api.dblab.dev))
     - Prometheus metrics endpoint (`/metrics`) for monitoring
     - Netdata module for insights
 
+## Clone major upgrade
+
+A single clone can be moved to a newer PostgreSQL major version without touching the rest of the
+instance, which makes it practical to test an upgrade against production-like data:
+
+```bash
+dblab clone upgrade --target-version 17 my-clone
+```
+
+The same action is available in the UI on the clone page and over the API as
+`POST /clone/{id}/upgrade`.
+
+The clone's Postgres is shut down cleanly, `pg_upgrade --link` converts its data directory in
+place, and the clone restarts on an image of the target major. Because `--link` hard-links the
+data files instead of copying them, the upgrade is fast and consumes almost no extra space. The
+request returns as soon as the clone enters the `UPGRADING` state; watch the clone status for the
+result.
+
+**Enabling it.** Set `provision.pgUpgradeImage` to an upgrade image matching the target major.
+When it is unset the feature is simply unavailable and the endpoint says so; nothing else about
+the instance changes.
+
+```yaml
+provision:
+  pgUpgradeImage: "postgresai/pg-upgrade:17"
+  pgUpgradeTimeout: 3h
+  pgUpgradePullTimeout: 1h
+```
+
+`pgUpgradeTimeout` bounds a single upgrade run; it defaults to three hours. When the budget runs
+out, the upgrade container is removed and the clone is put back on the version its data directory
+still holds, exactly as it would be after any other failure.
+
+`pgUpgradePullTimeout` bounds each of the two image pulls that precede it, and defaults to one
+hour. The upgrade image carries the target major plus the server packages of the four preceding
+ones, so the first upgrade on an instance downloads several gigabytes; pre-pulling both images
+makes the step a no-op. Both pulls happen while the clone is still serving traffic, so exceeding
+the budget leaves the clone untouched and running. The two budgets are separate on purpose — a
+slow pull must not eat into the time `pg_upgrade` gets once the clone is already stopped.
+
+An upgrade request may name an image explicitly instead of letting the engine substitute the major
+in the current tag. Set `provision.upgradeImageAllowList` to restrict which repositories such a
+request may name; when it is empty, as it is by default, any repository the instance can reach is
+accepted.
+
+```yaml
+provision:
+  upgradeImageAllowList:
+    - "postgresai/extended-postgres"
+```
+
+**How it can end.** Almost every outcome leaves the clone running:
+
+| Status | Meaning |
+|---|---|
+| `OK` | The clone runs the target version; `dbVersion` reports it. |
+| `WARNING` | The upgrade was not applied. Either nothing had been converted and the clone still runs its original version, or conversion had begun and the clone was rebuilt from its origin snapshot — in which case data written since the clone was created is lost. The message says which, and points at the `pg_upgrade` log under `<clone dir>/upgrade/logs/`. |
+| `WARNING`, clone not running | The upgrade settled on disk but the clone container did not come back up. The data directory is intact and the upgrade is still recorded as pending, so the next engine start finishes or undoes it and brings the clone back. |
+| `FATAL` | Recovery itself failed and there is nothing left to finish. Reset or delete the clone. |
+
+**Rolling back.** Resetting the clone returns it to the version the instance is configured with,
+because a reset re-provisions from the origin snapshot using the engine-wide image.
+
+**Things to know.**
+
+- The target must be newer than the clone's current major and at most four majors ahead — the
+  upgrade image carries binaries for the four preceding versions.
+- Only majors DBLab ships a default configuration for can be targeted (10–18 today).
+- The image tag keeps its extension bundle and glibc suffix; only the major changes
+  (`…:16-0.8.0-glibc236` → `…:17-0.8.0-glibc236`). Changing the glibc build across an upgrade
+  would change collation behaviour, and `pg_upgrade` does not reindex. Pass `--docker-image` to
+  override the choice.
+- `pg_upgrade` verifies that every extension in the source database has a matching library for
+  the new major. An extension outside the image's set fails this check, and the clone rolls back
+  untouched.
+- Clones using non-default tablespaces are refused: those live outside the clone dataset.
+- In physical mode the sync instance and the pool are untouched — the upgrade only ever acts on a
+  clone's own dataset. The instance-wide `dockerImage` must still match the source major.
+- **Iteration 1 limitation:** a snapshot created from an upgraded clone does not record its major
+  version, so clones created from that snapshot start on the instance-wide image and will fail to
+  start. Avoid snapshotting upgraded clones until snapshot-level version resolution lands.
+
 ## How to contribute
 ### Support us on GitHub/GitLab
 The simplest way to show your support is by giving us a star on GitHub or GitLab! ⭐