docs: add A23 runner-doctor failure mode for arc-dind credential overlay EROFS - #8020
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The generated workflow is stale, the portable playbook is incomplete, and the documented probe describes the wrong mount shape.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Documents A23, an ARC/DinD credential-overlay EROFS failure fixed by #7998.
Changes:
- Adds A23 catalog and lookup entries.
- Updates the workflow diagnostic playbook and portable agent.
- Adds alignment assertions.
File summaries
| File | Description |
|---|---|
.github/workflows/shared/self-hosted-failure-modes.md |
Adds canonical A23 documentation. |
.github/workflows/self-hosted-runner-doctor.md |
Adds A23 diagnostic matching. |
.github/agents/self-hosted-runner-doctor.md |
Mirrors A23 catalog content. |
scripts/ci/self-hosted-runner-doctor-workflow.test.ts |
Tests A23 catalog alignment. |
Review details
Suppressed comments (1)
.github/agents/self-hosted-runner-doctor.md:266
- This mirrored probe has the same incorrect mount description as the shared catalog: the failing overlay is
/dev/null:/host${effectiveHome}/<credential>:ro, whileworkDirbelongs to the separate prefixed chroot-home mount source. Update the probe to inspect those two surviving mounts rather than an overlay targeting an unprefixed home/workDir path.
| A23 | On `runner.topology: arc-dind` with `--docker-host-path-prefix` set, `docker compose up` fails with `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (also seen for `.docker/config.json`, `.composer/auth.json`) | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) is meant to drop the bogus AWF-owned chroot-home mount that the DinD daemon can't resolve, but it compared already-prefixed mount sources (from `buildAgentVolumes()`, which applies `--docker-host-path-prefix` as its final step) against the *raw, unprefixed* `config.workDir`/`effectiveHome`. Once a host-path prefix is set — always true on real ARC/DinD — the comparison silently stopped matching, so the bogus chroot-home mount survived filtering and Docker tried to create a `.npmrc` credential-hiding overlay mountpoint inside a path the daemon couldn't write to (EROFS). Distinct from A20 (which was about legitimate caller-supplied home mounts being dropped); A23 is the inverse case — the *bogus* mount not being dropped. | **Fixed in AWF (PR github/gh-aw-firewall#7998, merged 2026-09-02):** extracted `prefixHostPath()` in `src/services/host-path-prefix.ts` from `translateBindMountHostPath()` so bare paths can be prefixed consistently; `filterAgentVolumesForSysroot()` now prefixes `config.workDir`/`effectiveHome` before comparing against mount sources, restoring correct detection of daemon-invisible mounts. The existing safe fallback (`dropUnbackedHostHomeOverlays`, warn + skip masking) now engages correctly instead of silently failing; explicit writable `--mount` for the home root still preserves credential masking. Upgrade AWF to include github/gh-aw-firewall#7998. | Reproduce with `runnerTopology: 'arc-dind'` + `dockerHostPathPrefix` set (e.g. `/host`) and inspect generated compose for a `.npmrc`/`.docker/config.json`/`.composer/auth.json` `/dev/null` overlay mountpoint targeting an unprefixed `effectiveHome`/`workDir` path; on unpatched AWF, `docker compose up` fails with the EROFS mounting error above | github/gh-aw#57468, github/gh-aw-firewall#7994, github/gh-aw-firewall#7998 |
- Files reviewed: 4/4 changed files
- Comments generated: 4
- Review effort level: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244) | ||
| - `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21 (silent security-feature degradation under `filesystem.allowWrite`, fixed in github/gh-aw-firewall#7679) | ||
| - `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) | ||
| - `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (or `.docker/config.json`, `.composer/auth.json`) on `arc-dind` with `--docker-host-path-prefix` set → A23 (`filterAgentVolumesForSysroot()` compared prefixed mount sources against unprefixed `workDir`/`effectiveHome`, so the bogus chroot-home mount wasn't dropped; fixed in github/gh-aw-firewall#7998) |
| // A23 new failure mode (arc-dind /dev/null credential overlay EROFS with --docker-host-path-prefix) | ||
| expect(content).toContain('| A23 | On `runner.topology: arc-dind` with `--docker-host-path-prefix` set, `docker compose up` fails with `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system`'); | ||
| expect(content).toContain('`filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) is meant to drop the bogus AWF-owned chroot-home mount'); | ||
| expect(content).toContain('extracted `prefixHostPath()` in `src/services/host-path-prefix.ts`'); | ||
| expect(content).toContain('`filterAgentVolumesForSysroot()` now prefixes `config.workDir`/`effectiveHome` before comparing against mount sources'); | ||
| expect(content).toContain('github/gh-aw#57468, github/gh-aw-firewall#7994, github/gh-aw-firewall#7998'); | ||
| expect(content).toContain('| `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (or `.docker/config.json`, `.composer/auth.json`) on `arc-dind` with `--docker-host-path-prefix` set | A23'); |
| | A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 | | ||
| | A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2/<layer-id>/merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; inspect entrypoint logs for `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` — on older AWF this confirms the silent-degradation mode; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 | | ||
| | A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `src/services/squid-service.ts` and `src/services/agent-service.ts` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read daemon-side from `/proc/self/status` `CapBnd` via a privileged probe container) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `getHostCapabilityBoundingSet()` (`src/capability-filter.ts:78-89`) runs `docker run --rm --privileged --network=none alpine:latest cat /proc/self/status` against the daemon and decodes `CapBnd`; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | | ||
| | A23 | On `runner.topology: arc-dind` with `--docker-host-path-prefix` set, `docker compose up` fails with `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (also seen for `.docker/config.json`, `.composer/auth.json`) | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) is meant to drop the bogus AWF-owned chroot-home mount that the DinD daemon can't resolve, but it compared already-prefixed mount sources (from `buildAgentVolumes()`, which applies `--docker-host-path-prefix` as its final step) against the *raw, unprefixed* `config.workDir`/`effectiveHome`. Once a host-path prefix is set — always true on real ARC/DinD — the comparison silently stopped matching, so the bogus chroot-home mount survived filtering and Docker tried to create a `.npmrc` credential-hiding overlay mountpoint inside a path the daemon couldn't write to (EROFS). Distinct from A20 (which was about legitimate caller-supplied home mounts being dropped); A23 is the inverse case — the *bogus* mount not being dropped. | **Fixed in AWF (PR github/gh-aw-firewall#7998, merged 2026-09-02):** extracted `prefixHostPath()` in `src/services/host-path-prefix.ts` from `translateBindMountHostPath()` so bare paths can be prefixed consistently; `filterAgentVolumesForSysroot()` now prefixes `config.workDir`/`effectiveHome` before comparing against mount sources, restoring correct detection of daemon-invisible mounts. The existing safe fallback (`dropUnbackedHostHomeOverlays`, warn + skip masking) now engages correctly instead of silently failing; explicit writable `--mount` for the home root still preserves credential masking. Upgrade AWF to include github/gh-aw-firewall#7998. | Reproduce with `runnerTopology: 'arc-dind'` + `dockerHostPathPrefix` set (e.g. `/host`) and inspect generated compose for a `.npmrc`/`.docker/config.json`/`.composer/auth.json` `/dev/null` overlay mountpoint targeting an unprefixed `effectiveHome`/`workDir` path; on unpatched AWF, `docker compose up` fails with the EROFS mounting error above | github/gh-aw#57468, github/gh-aw-firewall#7994, github/gh-aw-firewall#7998 | |
| | A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 | | ||
| | A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2/<layer-id>/merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; inspect entrypoint logs for `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` — on older AWF this confirms the silent-degradation mode; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 | | ||
| | A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `src/services/squid-service.ts` and `src/services/agent-service.ts` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read daemon-side from `/proc/self/status` `CapBnd` via a privileged probe container) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `getHostCapabilityBoundingSet()` (`src/capability-filter.ts:78-89`) runs `docker run --rm --privileged --network=none alpine:latest cat /proc/self/status` against the daemon and decodes `CapBnd`; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | | ||
| | A23 | On `runner.topology: arc-dind` with `--docker-host-path-prefix` set, `docker compose up` fails with `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (also seen for `.docker/config.json`, `.composer/auth.json`) | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) is meant to drop the bogus AWF-owned chroot-home mount that the DinD daemon can't resolve, but it compared already-prefixed mount sources (from `buildAgentVolumes()`, which applies `--docker-host-path-prefix` as its final step) against the *raw, unprefixed* `config.workDir`/`effectiveHome`. Once a host-path prefix is set — always true on real ARC/DinD — the comparison silently stopped matching, so the bogus chroot-home mount survived filtering and Docker tried to create a `.npmrc` credential-hiding overlay mountpoint inside a path the daemon couldn't write to (EROFS). Distinct from A20 (which was about legitimate caller-supplied home mounts being dropped); A23 is the inverse case — the *bogus* mount not being dropped. | **Fixed in AWF (PR github/gh-aw-firewall#7998, merged 2026-09-02):** extracted `prefixHostPath()` in `src/services/host-path-prefix.ts` from `translateBindMountHostPath()` so bare paths can be prefixed consistently; `filterAgentVolumesForSysroot()` now prefixes `config.workDir`/`effectiveHome` before comparing against mount sources, restoring correct detection of daemon-invisible mounts. The existing safe fallback (`dropUnbackedHostHomeOverlays`, warn + skip masking) now engages correctly instead of silently failing; explicit writable `--mount` for the home root still preserves credential masking. Upgrade AWF to include github/gh-aw-firewall#7998. | Reproduce with `runnerTopology: 'arc-dind'` + `dockerHostPathPrefix` set (e.g. `/host`) and inspect generated compose for a `.npmrc`/`.docker/config.json`/`.composer/auth.json` `/dev/null` overlay mountpoint targeting an unprefixed `effectiveHome`/`workDir` path; on unpatched AWF, `docker compose up` fails with the EROFS mounting error above | github/gh-aw#57468, github/gh-aw-firewall#7994, github/gh-aw-firewall#7998 | |
|
@copilot address review feedback |
Addressed the review feedback in commit |
|
✅ Copilot review passed with no inline comments. @copilot Add the |
Add runner-doctor failure mode A23 documenting an arc-dind credential-overlay mounting error when
--docker-host-path-prefixis set. The issue occurs whenfilterAgentVolumesForSysroot()compares already-prefixed mount sources against unprefixedworkDir/effectiveHomepaths, causing the bogus chroot-home mount to survive filtering and Docker attempting to create/dev/nulloverlays inside a read-only daemon path (EROFS). The fix is already deployed (#7998, merged 2026-09-02); this PR documents the failure mode and fix in the runner-doctor knowledge base.Changes:
Shared failure-modes catalog (
.github/workflows/shared/self-hosted-failure-modes.md)Workflow playbook (
.github/workflows/self-hosted-runner-doctor.md)Portable agent documentation (
.github/agents/self-hosted-runner-doctor.md)Test suite (
scripts/ci/self-hosted-runner-doctor-workflow.test.ts)The three runner-doctor files are kept in sync by the updater workflow; this PR ensures new failure modes propagate correctly to all three locations.