docs: add A23 runner-doctor failure mode for arc-dind credential overlay EROFS (#8020)

* Initial plan

* chore: add A23 runner-doctor failure mode for arc-dind /dev/null overlay EROFS

* docs: address runner doctor A23 review feedback

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

Author: Copilot

.github/agents/self-hosted-runner-doctor.md

@@ -115,6 +115,7 @@ Prefer the narrowest match. Examples:
 - `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244)
 - `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21 (silent security-feature degradation under `filesystem.allowWrite`, fixed in github/gh-aw-firewall#7679)
 - `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795)
+- `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (or `.docker/config.json`, `.composer/auth.json`) on `arc-dind` with `--docker-host-path-prefix` set → A23 (a surviving prefixed `${workDir}-chroot-home:/host$HOME` mount is paired with `/dev/null:/host$HOME/<credential>:ro` overlays because prefixed sources were compared with unprefixed `workDir`/`effectiveHome`; fixed in github/gh-aw-firewall#7998)
 - `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817)
 - TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`)
 
@@ -263,6 +264,7 @@ Establish these facts before matching a failure mode:
 | A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 |
 | A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2/<layer-id>/merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; inspect entrypoint logs for `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` — on older AWF this confirms the silent-degradation mode; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 |
 | A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `src/services/squid-service.ts` and `src/services/agent-service.ts` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read daemon-side from `/proc/self/status` `CapBnd` via a privileged probe container) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `getHostCapabilityBoundingSet()` (`src/capability-filter.ts:78-89`) runs `docker run --rm --privileged --network=none alpine:latest cat /proc/self/status` against the daemon and decodes `CapBnd`; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 |
+| A23 | On `runner.topology: arc-dind` with `--docker-host-path-prefix` set, `docker compose up` fails with `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (also seen for `.docker/config.json`, `.composer/auth.json`) | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) is meant to drop the bogus AWF-owned chroot-home mount that the DinD daemon can't resolve, but it compared already-prefixed mount sources (from `buildAgentVolumes()`, which applies `--docker-host-path-prefix` as its final step) against the *raw, unprefixed* `config.workDir`/`effectiveHome`. Once a host-path prefix is set — always true on real ARC/DinD — the comparison silently stopped matching, so the bogus chroot-home mount survived filtering and Docker tried to create a `.npmrc` credential-hiding overlay mountpoint inside a path the daemon couldn't write to (EROFS). Distinct from A20 (which was about legitimate caller-supplied home mounts being dropped); A23 is the inverse case — the *bogus* mount not being dropped. | **Fixed in AWF (PR github/gh-aw-firewall#7998, merged 2026-09-02):** extracted `prefixHostPath()` in `src/services/host-path-prefix.ts` from `translateBindMountHostPath()` so bare paths can be prefixed consistently; `filterAgentVolumesForSysroot()` now prefixes `config.workDir`/`effectiveHome` before comparing against mount sources, restoring correct detection of daemon-invisible mounts. The existing safe fallback (`dropUnbackedHostHomeOverlays`, warn + skip masking) now engages correctly instead of silently failing; explicit writable `--mount` for the home root still preserves credential masking. Upgrade AWF to include github/gh-aw-firewall#7998. | Reproduce with `runnerTopology: 'arc-dind'` + `dockerHostPathPrefix` set (e.g. `/host`) and inspect generated compose for a `.npmrc`/`.docker/config.json`/`.composer/auth.json` `/dev/null` overlay mountpoint targeting an unprefixed `effectiveHome`/`workDir` path; on unpatched AWF, `docker compose up` fails with the EROFS mounting error above | github/gh-aw#57468, github/gh-aw-firewall#7994, github/gh-aw-firewall#7998 |
 
 ## Category B — Self-hosted runners
 
@@ -385,6 +387,7 @@ Establish these facts before matching a failure mode:
 | `Host workspace is not writable by the sandbox identity (<uid>:<gid>): <path>` | B25 |
 | `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode | B26 (mcpg stopped auto-following the artifact redirect; cli-proxy had no route to Azure Blob storage; fixed in github/gh-aw-firewall#7635 with scoped Squid ACL keyed to cli-proxy's fixed IP) |
 | `invalid CapDrop: capability not supported by your kernel or not available in the current environment` | A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) |
+| `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (or `.docker/config.json`, `.composer/auth.json`) on `arc-dind` with `--docker-host-path-prefix` set | A23 (`filterAgentVolumesForSysroot()` compared prefixed mount sources against unprefixed `workDir`/`effectiveHome`, so the bogus chroot-home mount wasn't dropped; fixed in github/gh-aw-firewall#7998) |
 | `a network with name awf-net exists but was not created for project` | B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) |
 | TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA | B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) |
 | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) |