feat(cli): rework build / test environment handling

Adds an explicit environment (env) subcommand which allows users
to list and interact with existing build / test environments.

Additionally contains extensive internal refactoring to make the
environment tracking and cleanup more robust.

Author: mikonse

.github/workflows/ci.yaml

@@ -141,6 +141,10 @@ jobs:
       contents: read
       # Push of the pre-built CI image is additionally gated to main below.
       packages: write
+    env:
+      # Registry and Environment Host roots together, so the cached build
+      # tree stays registered under the same Environment id.
+      DEBMAGIC_DATA_DIR: /tmp/debmagic-data
     steps:
       - uses: actions/checkout@v5
       - uses: Swatinem/rust-cache@v2
@@ -149,16 +153,18 @@ jobs:
       - name: Cache the debmagic build tree
         uses: actions/cache@v4
         with:
-          path: /tmp/debmagic
-          key: debmagic-build-${{ hashFiles('Cargo.lock', 'debian/changelog', 'debian/control', 'debian/rules') }}
-          restore-keys: debmagic-build-
-      - name: Prune stale build trees from the restored cache
+          path: /tmp/debmagic-data
+          key: debmagic-data-${{ hashFiles('Cargo.lock', 'debian/changelog', 'debian/control', 'debian/rules') }}
+          restore-keys: debmagic-data-
+      - name: Prune Environments of other package versions from the restored cache
         run: |
-          mkdir -p /tmp/debmagic
           current="$(dpkg-parsechangelog -SSource)-$(dpkg-parsechangelog -SVersion)"
-          find /tmp/debmagic -mindepth 1 -maxdepth 1 -type d \
-            ! -name "$current" ! -name "${current}-test" \
-            -exec rm -rf {} +
+          cargo run --locked -q -p debmagic -- env list \
+            | awk -v current="$current" 'NR > 1 && $4 != current { print $1 }' \
+            | while read -r id; do
+                cargo run --locked -q -p debmagic -- env clean "$id" \
+                  || echo "::warning::could not prune Environment $id"
+              done
       - name: Resolve CI image tag
         id: image
         run: |

Cargo.lock

@@ -35,6 +35,7 @@ ignore = { version = ">=0.4.25" }
 md-5 = { version = ">=0.11" }
 sha1 = { version = ">=0.11" }
 sha2 = { version = ">=0.11" }
+rusqlite = { version = ">=0.32" }
 
 # dev dependencies
 test-case = { version = ">=3.3.1" }

Cargo.toml

@@ -7,7 +7,7 @@ Modern, robust & easy [Debian](https://debian.org)/[Ubuntu](https://ubuntu.com)
 Debmagic unifies the the packaging experience as a single, streamlined tool. Additionally, `debmagic-pkg` allows you to define package build recipes in Python.
 
 - **Isolated builds without the setup**: `debmagic build binary` builds a `debian/`-packaged source tree in a temporary container (using LXD, Incus, Docker)
-- **Fast iteration**: `--persistent`/`--incremental` reuse the environment and sync only source changes; `--shell-on-failure` and `debmagic shell` drop you right where the build broke
+- **Fast iteration**: `--persistent`/`--incremental` reuse the environment and sync only source changes; `--shell-on-failure` and `debmagic env shell` drop you right where the build broke
 - **Test & sign integrated**: `debmagic test` runs the package's autopkgtest tests in a fresh environment, `debmagic sign` GPG-signs `.changes`/`.dsc`/`.buildinfo` on the host
 - **Python packaging API**: replace complicated `debian/rules` Makefiles with typed Python `debian/rules.py`, with optional `dh` compatibility
 
@@ -30,7 +30,7 @@ uvx debmagic build binary --driver docker
 | `debmagic build binary` | Build a binary package (`.deb`) in an isolated environment |
 | `debmagic build source` | Create a source package (`.dsc`) for upload (incl signing) |
 | `debmagic test` | Run the package's autopkgtest tests (`debian/tests/`) against a prior build |
-| `debmagic shell` | Attach an interactive shell to the build environment |
+| `debmagic env` | List, clean, or open a shell in a build or test environment |
 | `debmagic sign` | GPG-sign a `.changes` file (and its `.dsc`/`.buildinfo`) on the host |
 | `debmagic config` | Inspect and edit the effective `debmagic.toml` configuration |
 | `debmagic check` | Lint the package *(in progress)* |

README.md

@@ -38,6 +38,9 @@ Build-Depends:
  librust-md-5-dev (>= 0.11),
  librust-sha1-dev (>= 0.11),
  librust-sha2-dev (>= 0.11),
+ librust-rusqlite-dev (>= 0.32),
+ libsqlite3-dev,
+ pkg-config,
 Rules-Requires-Root: no
 X-Style: black
 Standards-Version: 4.7.2

debian/control

@@ -17,13 +17,43 @@
 )
 
 # TODO: add this section to our rust debmagic module to make setting this up less annoying
-os.environ.update({
-    "PATH": f"/usr/share/cargo/bin:{os.environ['PATH']}",
-    "CARGO": "/usr/share/cargo/bin/cargo",
-    "CARGO_HOME": f"{pkg.base_dir}/debian/cargo_home",
-    "CARGO_REGISTRY": f"{pkg.base_dir}/debian/cargo_registry",
-    "DEB_CARGO_CRATE": f"{pkg.build_env.DEB_SOURCE}_{pkg.build_env.DEB_VERSION_UPSTREAM}"
-})
+os.environ.update(
+    {
+        "PATH": f"/usr/share/cargo/bin:{os.environ['PATH']}",
+        "CARGO": "/usr/share/cargo/bin/cargo",
+        "CARGO_HOME": f"{pkg.base_dir}/debian/cargo_home",
+        "CARGO_REGISTRY": f"{pkg.base_dir}/debian/cargo_registry",
+        "DEB_CARGO_CRATE": f"{pkg.build_env.DEB_SOURCE}_{pkg.build_env.DEB_VERSION_UPSTREAM}",
+        # debmagic-pkg is setuptools and ignores this. The rewrite below is what
+        # makes resolution succeed; this only keeps maturin from enabling the
+        # (now empty) default feature.
+        "MATURIN_PEP517_ARGS": "--no-default-features",
+    }
+)
+
+# Cargo resolves every feature name in the table, including ones that are not
+# enabled, so `--no-default-features` still requires rusqlite to advertise
+# `bundled`. Debian's librust-rusqlite-dev deletes that feature and links the
+# system libsqlite3. Drop the reference for this build, then put the manifest
+# back so a host or incremental tree is not left edited.
+cargo_toml = Path("packages/debmagic/Cargo.toml")
+bundled_sqlite = 'bundled-sqlite = ["rusqlite/bundled"]'
+system_sqlite = "bundled-sqlite = []"
+
+
+def use_system_sqlite() -> str:
+    original = cargo_toml.read_text()
+    if bundled_sqlite in original:
+        cargo_toml.write_text(original.replace(bundled_sqlite, system_sqlite, 1))
+        return original
+    if system_sqlite in original:
+        restored = original.replace(system_sqlite, bundled_sqlite, 1)
+        return restored
+    raise SystemExit(
+        f"{cargo_toml}: expected {bundled_sqlite!r} so the Debian build can drop "
+        "the rusqlite bundled feature that librust-rusqlite-dev removes"
+    )
+
 
 packages = {
     "debmagic": ("packages/debmagic", "debmagic"),
@@ -33,10 +63,14 @@
 cargo_lock = Path("Cargo.lock")
 cargo_lock_saved = Path("Cargo.lock.saved")
 
+
 def dh_auto(build: Build, stage: str, use_destdir: bool = False):
     for pkg_name, (path, python_pkg_name) in packages.items():
         destdir = f" --destdir debian/{pkg_name} " if use_destdir else ""
-        build.cmd(f"{stage} -p {pkg_name} --sourcedirectory {path} --buildsystem=pybuild {destdir} -- --name {python_pkg_name}")
+        build.cmd(
+            f"{stage} -p {pkg_name} --sourcedirectory {path} --buildsystem=pybuild {destdir} -- --name {python_pkg_name}"
+        )
+
 
 @dhp.override
 def dh_auto_configure(build: Build):
@@ -68,4 +102,8 @@ def dh_auto_clean(build: Build):
     dh_auto(build, "dh_auto_clean")
 
 
-pkg.pack()
+cargo_toml_original = use_system_sqlite()
+try:
+    pkg.pack()
+finally:
+    cargo_toml.write_text(cargo_toml_original)

debian/rules.py

@@ -9,7 +9,7 @@ Working with Debian packages without `debmagic` means juggling with several inde
 Debmagic unifies this into one CLI with useful defaults:
 
 - **Isolated builds without the setup**: `debmagic build binary` builds a `debian/`-packaged source tree in a temporary container (using LXD, Incus, Docker)
-- **Fast iteration**: `--persistent`/`--incremental` reuse the environment and sync only source changes; `--shell-on-failure` and `debmagic shell` drop you right where the build broke
+- **Fast iteration**: `--persistent`/`--incremental` reuse the environment and sync only source changes; `--shell-on-failure` and `debmagic env shell` drop you right where the build broke
 - **Test & sign integrated**: `debmagic test` runs the package's autopkgtest tests in a fresh environment, `debmagic sign` GPG-signs `.changes`/`.dsc`/`.buildinfo` on the host
 - **Package uploading**: `debmagic upload` (to `debian`, `ubuntu`, `ppa:user/repo`) pushes signed builds to archives and PPAs via scp/sftp, with configurable pre-upload checks
 - **Upstream tracking**: `debmagic upstream list`/`switch` replace uscan/uupdate, reading the package's existing `debian/watch` and `debian/copyright`
@@ -37,6 +37,7 @@ Then dive into the pages below — start with [Getting started](usage/getting-st
 usage/getting-started.md
 usage/build.md
 usage/test.md
+usage/env.md
 usage/source.md
 usage/upload.md
 usage/upstream.md

docs/index.md

@@ -36,7 +36,7 @@ debmagic build binary --driver lxd \
 | `--output-dir <dir>` | Directory to put the resulting build artifacts |
 | `--shell-on-failure` | On build failure, drop into an interactive shell in the build environment when stdout is a TTY |
 
-[`debmagic shell`](#inspecting-a-failed-build) — attach an interactive shell to a build environment
+[`debmagic env shell`](#inspecting-a-failed-build) — attach an interactive shell to an Environment
 
 ## Picking a driver
 
@@ -54,13 +54,14 @@ There's no auto-detection; pick one and pass it explicitly every time (or config
 
 On failure the build environment is torn down by default. Pass `--shell-on-failure` to drop into an interactive shell inside the build environment when stdout is a TTY (destroyed on shell exit unless `--persistent` was used).
 
-To inspect after the run finishes, pass `--persistent` up front, then:
+To inspect after the run finishes, pass `--persistent` up front, then use [`debmagic env shell`](env.md).
 
 ```shell
-# if you're in the package still
-debmagic shell
-# from the outside:
-debmagic shell --source-dir /path/to/parent/of/debian/dir
+# if you're in the package still and exactly one Environment exists for it
+debmagic env shell
+# from anywhere:
+debmagic env list
+debmagic env shell <id>
 ```
 
 This attaches an interactive shell inside the still-running (or restartable) build environment, at the package's build directory.
@@ -129,7 +130,7 @@ This flag implies `--persistent`, and cannot be combined with `--clean yes`.
 The preserved build tree is kept even when the environment itself is *not* reused (e.g. a fresh CI runner where the tree was restored from a cache).
 Quilt patches that a previous build or shell session left applied in the build tree are unapplied with `dpkg-source --after-build` before sources are synced, so the worktree remains the source of truth and stale `.pc` state cannot break later builds.
 
-The work dir (`temp_build_dir`, default `/tmp/debmagic`) keeps the artifacts of builds, including their `.changes` files.
+The Environment's Host root lives under `environments_dir` (default `$XDG_DATA_HOME/debmagic/environments`). Build artifacts, including `.changes` files, are exported to `output_dir`.
 
 
 ## Selecting a distro/release
@@ -234,4 +235,4 @@ Instead of repeating CLI flags on every invocation, drop a [config file](config.
 
 ## Internals
 
-- Container/device names are derived and sanitized internally (alphanumeric + hyphen, ≤63 chars for LXD/Incus) — don't try to predict or construct them yourself; use `debmagic shell` instead of `lxc`/`docker` commands directly.
+- Container/device names are derived and sanitized internally (alphanumeric + hyphen, ≤63 chars for LXD/Incus) — don't try to predict or construct them yourself; use `debmagic env shell` instead of `lxc`/`docker` commands directly.