feat(cli): rework build / test environment handling

Adds an explicit environment (env) subcommand which allows users
to list and interact with existing build / test environments.

Additionally contains extensive internal refactoring to make the
environment tracking and cleanup more robust.

Author: mikonse

.github/workflows/ci.yaml

@@ -141,6 +141,10 @@ jobs:
       contents: read
       # Push of the pre-built CI image is additionally gated to main below.
       packages: write
+    env:
+      # Registry and Environment Host roots together, so the cached build
+      # tree stays registered under the same Environment id.
+      DEBMAGIC_DATA_DIR: /tmp/debmagic-data
     steps:
       - uses: actions/checkout@v5
       - uses: Swatinem/rust-cache@v2
@@ -149,16 +153,18 @@ jobs:
       - name: Cache the debmagic build tree
         uses: actions/cache@v4
         with:
-          path: /tmp/debmagic
-          key: debmagic-build-${{ hashFiles('Cargo.lock', 'debian/changelog', 'debian/control', 'debian/rules') }}
-          restore-keys: debmagic-build-
-      - name: Prune stale build trees from the restored cache
+          path: /tmp/debmagic-data
+          key: debmagic-data-${{ hashFiles('Cargo.lock', 'debian/changelog', 'debian/control', 'debian/rules') }}
+          restore-keys: debmagic-data-
+      - name: Prune Environments of other package versions from the restored cache
         run: |
-          mkdir -p /tmp/debmagic
           current="$(dpkg-parsechangelog -SSource)-$(dpkg-parsechangelog -SVersion)"
-          find /tmp/debmagic -mindepth 1 -maxdepth 1 -type d \
-            ! -name "$current" ! -name "${current}-test" \
-            -exec rm -rf {} +
+          cargo run --locked -q -p debmagic -- env list \
+            | awk -v current="$current" 'NR > 1 && $4 != current { print $1 }' \
+            | while read -r id; do
+                cargo run --locked -q -p debmagic -- env clean "$id" \
+                  || echo "::warning::could not prune Environment $id"
+              done
       - name: Resolve CI image tag
         id: image
         run: |

Cargo.lock

@@ -32,6 +32,7 @@ ignore = { version = ">=0.4.25" }
 md-5 = { version = ">=0.11" }
 sha1 = { version = ">=0.11" }
 sha2 = { version = ">=0.11" }
+rusqlite = { version = ">=0.32" }
 
 # dev dependencies
 test-case = { version = ">=3.3.1" }

Cargo.toml

@@ -10,7 +10,7 @@ Unified and future-proof developer tooling for increased productivity in the [De
 Debmagic has two independent parts:
 - tooling to [**build** and **test**](#debmagic-building) packages
   - in isolated container environments (lxd, docker, ...)
-  - `debmagic build`, `debmagic check`, `debmagic test`, `debmagic shell`, ...
+  - `debmagic build`, `debmagic check`, `debmagic test`, `debmagic env`, ...
 - create package [**build instructions**](#debmagic-packaging)
   - using Python with `debian/rules.py` (instead of shell & Makefiles)
 

README.md

@@ -34,7 +34,10 @@ Build-Depends:
  librust-ignore-dev (>=0.4.25),
  librust-md-5-dev (>= 0.11),
  librust-sha1-dev (>= 0.11),
- librust-sha2-dev (>= 0.11)
+ librust-sha2-dev (>= 0.11),
+ librust-rusqlite-dev (>= 0.32),
+ libsqlite3-dev,
+ pkg-config
 Rules-Requires-Root: no
 X-Style: black
 Standards-Version: 4.7.2

debian/control

@@ -22,9 +22,36 @@ os.environ.update({
     "CARGO": "/usr/share/cargo/bin/cargo",
     "CARGO_HOME": f"{pkg.base_dir}/debian/cargo_home",
     "CARGO_REGISTRY": f"{pkg.base_dir}/debian/cargo_registry",
-    "DEB_CARGO_CRATE": f"{pkg.build_env.DEB_SOURCE}_{pkg.build_env.DEB_VERSION_UPSTREAM}"
+    "DEB_CARGO_CRATE": f"{pkg.build_env.DEB_SOURCE}_{pkg.build_env.DEB_VERSION_UPSTREAM}",
+    # debmagic-pkg is setuptools and ignores this. The rewrite below is what
+    # makes resolution succeed; this only keeps maturin from enabling the
+    # (now empty) default feature.
+    "MATURIN_PEP517_ARGS": "--no-default-features",
 })
 
+# Cargo resolves every feature name in the table, including ones that are not
+# enabled, so `--no-default-features` still requires rusqlite to advertise
+# `bundled`. Debian's librust-rusqlite-dev deletes that feature and links the
+# system libsqlite3. Drop the reference for this build, then put the manifest
+# back so a host or incremental tree is not left edited.
+cargo_toml = Path("packages/debmagic/Cargo.toml")
+bundled_sqlite = 'bundled-sqlite = ["rusqlite/bundled"]'
+system_sqlite = "bundled-sqlite = []"
+
+
+def use_system_sqlite() -> str:
+    original = cargo_toml.read_text()
+    if bundled_sqlite in original:
+        cargo_toml.write_text(original.replace(bundled_sqlite, system_sqlite, 1))
+        return original
+    if system_sqlite in original:
+        restored = original.replace(system_sqlite, bundled_sqlite, 1)
+        return restored
+    raise SystemExit(
+        f"{cargo_toml}: expected {bundled_sqlite!r} so the Debian build can drop "
+        "the rusqlite bundled feature that librust-rusqlite-dev removes"
+    )
+
 packages = {
     "debmagic": ("packages/debmagic", "debmagic"),
     "debmagic-pkg": ("packages/debmagic-pkg", "debmagic-pkg"),
@@ -68,4 +95,8 @@ def dh_auto_clean(build: Build):
     dh_auto(build, "dh_auto_clean")
 
 
-pkg.pack()
+cargo_toml_original = use_system_sqlite()
+try:
+    pkg.pack()
+finally:
+    cargo_toml.write_text(cargo_toml_original)

debian/rules

@@ -9,6 +9,7 @@
 usage/getting-started.md
 usage/build.md
 usage/test.md
+usage/env.md
 usage/source.md
 usage/config.md
 usage/modules/index.md

docs/index.md

@@ -34,7 +34,7 @@ debmagic build binary --driver lxd \
 | `--output-dir <dir>` | Directory to put the resulting build artifacts |
 | `--shell-on-failure` | On build failure, drop into an interactive shell in the build environment when stdout is a TTY |
 
-[`debmagic shell`](#inspecting-a-failed-build) — attach an interactive shell to a build environment
+[`debmagic env shell`](#inspecting-a-failed-build) — attach an interactive shell to an Environment
 
 ## Picking a driver
 
@@ -52,13 +52,14 @@ There's no auto-detection; pick one and pass it explicitly every time (or config
 
 On failure the build environment is torn down by default. Pass `--shell-on-failure` to drop into an interactive shell inside the build environment when stdout is a TTY (destroyed on shell exit unless `--persistent` was used).
 
-To inspect after the run finishes, pass `--persistent` up front, then:
+To inspect after the run finishes, pass `--persistent` up front, then use [`debmagic env shell`](env.md).
 
 ```shell
-# if you're in the package still
-debmagic shell
-# from the outside:
-debmagic shell --source-dir /path/to/parent/of/debian/dir
+# if you're in the package still and exactly one Environment exists for it
+debmagic env shell
+# from anywhere:
+debmagic env list
+debmagic env shell <id>
 ```
 
 This attaches an interactive shell inside the still-running (or restartable) build environment, at the package's build directory.
@@ -188,4 +189,4 @@ Instead of repeating CLI flags on every invocation, drop a [config file](config.
 
 ## Internals
 
-- Container/device names are derived and sanitized internally (alphanumeric + hyphen, ≤63 chars for LXD/Incus) — don't try to predict or construct them yourself; use `debmagic shell` instead of `lxc`/`docker` commands directly.
+- Container/device names are derived and sanitized internally (alphanumeric + hyphen, ≤63 chars for LXD/Incus) — don't try to predict or construct them yourself; use `debmagic env shell` instead of `lxc`/`docker` commands directly.

docs/usage/build.md

@@ -14,6 +14,8 @@ Config files are merged in order of increasing precedence, so later files overri
 Only files that exist are read; missing ones are skipped.
 Command-line flags override whatever the merged config resolves to.
 
+`DEBMAGIC_DATA_DIR` relocates the Environment registry (`db.sqlite`) and the default `environments_dir` together. A relative value is resolved to an absolute path against the current working directory. It is not a config file key.
+
 ## Options
 
 All keys are optional.
@@ -27,7 +29,7 @@ All keys are optional.
 | `driver.docker.base_images` | map | — | — | Base image per distro, keyed by `"<distro>:<codename>"` (e.g. `"debian:trixie"`). Falls back to `docker.io/<distro>:<codename>`. For non-Debian/Ubuntu suites (e.g. `"yocto:kirkstone"`), the map entry is what makes the suite a known DistroVersion for Docker builds. |
 | `driver.lxd.project` | string | — | — | LXD/Incus project to use. |
 | `driver.lxd.base_images` | map | — | — | Base image per distro, keyed by `"<distro>:<codename>"`. Falls back to the driver's default remote image. Same custom-suite registry role as Docker's map for LXD/Incus. |
-| `temp_build_dir` | path | `/tmp/debmagic` | — | Where build trees are staged. |
+| `environments_dir` | path | `$XDG_DATA_HOME/debmagic/environments` | — | Parent directory of Environment Host roots. Must be absolute or start with `~/`. The Environment registry is always `$XDG_DATA_HOME/debmagic/db.sqlite` (override both with `DEBMAGIC_DATA_DIR`). |
 | `output_dir` | path | `build` | `-o`/`--output-dir` | Where build artifacts are exported; relative paths resolve against the package root. |
 | `incremental` | bool | `false` | `--incremental` | Retain the environment and sync only source changes, preserving generated files. Binary-only; implies `persistent`; incompatible with `clean`. |
 | `source_sync_mode` | enum | `tracked` | `--source-sync` | Which source files are staged (see below). |