SECRETS-ENV-VAR-RISK.md

Container Env-Var Introspection Risk (Audit C-MED-4)

This is a design note about how Containarium delivers tenant secrets into containers, the risk that comes with the env-var path, and the tmpfs-mount (file delivery) alternative that closes it.

Today: file delivery is the default; env-var stamping remains available (#1604)

A SetSecret with no delivery specified now gets file — the tmpfs-mount path described below — rather than env-var stamping. Existing rows keep whatever mode they were written with; nothing is migrated retroactively, and a value rotation that doesn't repeat an explicit --delivery preserves the row's current mode rather than silently reclassifying it.

env remains fully supported and selectable via containarium secrets set <user> <NAME> <value> --delivery env — some apps only read os.Getenv and nothing else, and this document's risk analysis of that path is unchanged; it's just no longer what a caller gets without asking for it.

When env delivery is in effect, internal/server/secrets_server.go:stampSecretsOnLXC calls incus config set environment.<NAME>=<value> for that secret. The value lands in the LXC's Incus config; on the next container start (or RefreshSecrets call) it becomes an environment variable of the container's init process (PID 1) and inherits into every child process via execve(2).

The app code reads it as os.Getenv("OPENAI_API_KEY") or whatever's idiomatic in its language — convenient, language-agnostic, no SDK, and the reason env remains available rather than being removed.

What "in env" means in practice

Once a value is in the container's environment, every process that can escalate to the same UID — and many that can't — can read it:

Reader	Has access?
Any process running as the app UID	Yes (/proc/self/environ)
Root inside the container	Yes (/proc/<any-pid>/environ)
ps eww from a peer process	Yes if same UID, no otherwise
Container init's child after fork	Yes (inherited)
New exec inside the container	Yes
Docker / podman-in-LXC sub-container	Only if the operator explicitly forwards it
Operator on the daemon host	Yes (incus config show)
Another tenant's container	No (separate LXC, separate kernel namespaces)

The interesting columns for tenants are the first three. If an attacker gets RCE inside the container, even as a non-root user, they can read every secret the app process can read. The env vars are also visible in /proc/<pid>/environ to root — so a privileged daemon inside the same container (logging agent, supervisor) sees them too.

That's a wider surface than many operators assume. It does NOT cross tenant boundaries (one container can't read another's env), but it does NOT confine secrets within a single container either.

Threat model — what this DOES and DOES NOT protect

What's already protected (today):

• Cross-tenant exposure: secrets are scoped per username in the secrets table, encrypted with AES-256-GCM at rest, and stamped into exactly one LXC. Another tenant's container can't read them.

  On the API side this is auth.AuthorizeSecretTenant, which is stricter than the AuthorizeTenant used elsewhere: reaching another tenant's secrets needs the admin role and secrets:read / secrets:write granted explicitly on the token. The admin role alone does not imply it, because a token minted --roles admin with no --scopes — the ordinary shape for an orchestrator or automation account — otherwise passed both gates for every tenant. Automation that does not need cross-tenant secret access should simply not carry the scope.

• Operator audit trail: every set_secret / get_secret is audit-logged.

• At-rest encryption: the master key lives at /etc/containarium/secrets.key mode 0400, off the wire and not in any backup unless the operator explicitly arranges it.

What's NOT protected (audit C-MED-4):

• Same-container introspection: any process inside the container that can read /proc/<pid>/environ sees every secret. That's the Linux env-var contract.
• Docker-in-LXC sub-containers: the LXC has the env, but docker run / docker compose require explicit -e passthrough. Operators who forget the passthrough get a "missing env var" runtime error — visible failure mode, not silent.

Mitigations available today

For an operator who needs to harden this further without waiting for the tmpfs alternative:

1. Don't put high-risk secrets in env vars. Use the secret-manager pattern: store a short-lived bootstrap token in env, have the app fetch the real secret from a vault on startup. The bootstrap token stays in env; the real secret never does.
2. Drop privileges inside the container. A non-root app user with setuid semantics can read its own /proc/self/environ but not that of root daemons. (The reverse is also true — root can read everyone.)
3. Audit incus config show access. Operators who can run that command see every container's secrets. Limit who has the incus-admin group on the daemon host.
4. Rotate aggressively. The combination of Phase 1.2 (jti + revocation), Phase 1.6 (short-lived access + refresh rotation), and containarium secret set (which versions in place) means an exposed secret has a bounded blast radius.

The tmpfs-mount alternative (file delivery — shipped, now the default)

The cleanest mitigation is to stop putting secrets in env entirely:

• Mount a tmpfs into the container at e.g. /run/secrets/.
• Write each decrypted secret to its own file, mode 0400, owned by the app user (not root).
• App reads from the file instead of os.Getenv.

Properties:

• Per-process access control: file permissions, not the env-var free-for-all. The app user can read; root can read; nobody else.
• No execve inheritance: values aren't passed to child processes unless the app explicitly hands them off.
• No incus config show leak: tmpfs contents aren't in the Incus config — only the mount point is.
• Crash-safe disposal: tmpfs exists in kernel memory, evicted when the container stops. No risk of secrets persisting in container snapshots.

Costs:

• App-side change: read files instead of env. Most apps that already support <SECRET>_FILE env-var conventions (Postgres password, Vault sidecars, Docker Compose secrets) work with no code change — set PGPASSWORD_FILE=/run/secrets/PGPASSWORD instead of PGPASSWORD=…. Apps that only read os.Getenv need --delivery env instead — see the "Today" section above.
• One more code path on the daemon (mount management, file rotation on RefreshSecrets).

This ships alongside the env-var path, not as a replacement — env stays fully supported and selectable per-tenant or per-secret via --delivery env. It's the default (#1604) rather than opt-in, because the risk this section describes applies whether or not an operator knew to ask for the alternative.

Status: shipped through Phase B-2, default flipped in #1604

• Phase A (PR #274): the delivery field is plumbed through proto / Store / CLI. Operators can set it via containarium secrets set <user> <NAME> <value> --delivery file.
• Phase B-1 (PR #275): stampSecretsOnLXC dispatches per-secret. delivery="file" rows are written to /run/secrets/<NAME> mode 0400. Since /run is tmpfs on every systemd distro, file-mode secrets inherit the in-memory ephemeral-disposal property for free — the tmpfs evaporates when the container stops.
• Phase B-2 (this PR): tenant ownership. The /run/secrets directory is now 0750 root:<username> and each file is 0440 root:<username>. The app process — running as the tenant user inside the container, the convention since CreateContainer — can read its own secrets without sudo. Fallback: if the chown fails (early-boot race, user not in /etc/passwd), the file stays mode 0400 root and the operator sees a WARNING log line.

Wire shape inside the container:

$ ls -ld /run/secrets
drwxr-x---. 2 root alice 60 /run/secrets

$ ls -l /run/secrets
-r--r-----. 1 root alice  43 OPENAI_API_KEY
-r--r-----. 1 root alice 128 DATABASE_URL

The app running as alice can cat /run/secrets/OPENAI_API_KEY; no other in-container user can.

• Phase B-3 (this PR): periodic reconciler. The daemon ticks every 60s, asks the Store which tenants have file-mode secrets, and re-stamps each one whose container is Running. A bare incus restart not routed through the daemon now self-heals within the tick interval. Skipped on every tick:
  • tenants with no file-mode secrets (env-mode rows survive restart natively via incus config);
  • containers that are Stopped (next StartContainer will re-stamp).

The reconciler is owned by the daemon alongside the autosleep manager; same shape, same lifetime. Stamp is idempotent (writing the same content over the same path is a no-op at the bytes level), so periodic re-stamps cost nothing when state is already correct.

compose delivery — reaching nested docker / docker-compose

env and file delivery both miss one common app shape: a docker-compose stack running inside the LXC. A docker container does not inherit the LXC's Incus-config environment, and it doesn't see /run/secrets/<NAME> unless that path is explicitly mounted in. This is the same gap OTel monitoring hit (#370) and solved with a dotenv file consumed via env_file:.

--delivery compose closes it for secrets the same way:

• The daemon renders the tenant's compose-mode secrets into a single dotenv file at /run/containarium/secrets.env, mode 0400 root, on /run (tmpfs — evicted on stop, never on the writable rootfs).
• A compose service references it: env_file: /run/containarium/secrets.env. The compose runtime (root) reads the file at container-create time and injects the values as env into the app container — the app process never reads the file itself, so 0400 root is both safe and sufficient.

Exposure profile vs the other modes:

• vs env: does NOT leak via incus config show (the dotenv file isn't in the Incus config). Same in-container exposure once injected.
• vs file: one shared file instead of per-secret files; meant for compose ingestion, not per-process file ACLs.

Constraint: values must be single-line. A dotenv KEY=value line can't represent a newline, and the env_file: parser would mangle it — so a multi-line value is rejected at containarium secrets set --delivery compose time. Use --delivery file (/run/secrets/<NAME>) for multi-line blobs.

Built on the shared EnvFile delivery helper (#492); see pkg/core/container/{envfile,secrets_envfile}.go and the compose branch in stampSecretsOnLXC.

References

• Audit finding C-MED-4 in ZERO-TRUST-AUDIT.md.
• Implementation in internal/server/secrets_server.go.
• Related: PR #248 (jti + revocation), PRs #254 / #255 (refresh rotation) — both shrink the blast radius of any exposed credential derived from an env-var leak.