fix: devcontainer startup for ssh sessions - #2752
Draft
Rtosshy wants to merge 2 commits into
Draft
Conversation
…er start postAttachCommand only fires when an editor client attaches, so sessions opened with `gh codespace ssh` never ran setup.sh. That left the container without a Docker daemon and with root-owned cache volumes, which surfaced much later as unrelated-looking failures: Cannot connect to the Docker daemon at unix:///var/run/docker.sock open /go/pkg/sumdb/sum.golang.org/latest: no such file or directory error Error: EACCES: permission denied, mkdir '~/.yarn/cache/v6' Move both concerns out of setup.sh into scripts that postStartCommand runs at every container start, one concern each. setup.sh still calls them -- they are preconditions for its own work (yarn/go write into those volumes, cleanup_docker_if_needed shells out to docker) and both are no-ops once satisfied. That call also moves to the top of main(), so the daemon is up before cleanup_docker_if_needed uses it. Starting the daemon via `service docker start` keeps its log at /var/log/docker.log; the previous `nohup sudo dockerd > /tmp/dockerd.log` left nothing behind once /tmp was cleared. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…emon Codespaces runs the dev container in the host network namespace, so the Codespaces VM's Docker daemon and our nested one write firewall rules into the same namespace. The outer daemon is an older release on iptables-legacy: it sets `FORWARD policy DROP` there and opens holes only for its own docker0, which nothing is even attached to under host networking. Our daemon defaults to the nft backend, so minikube's bridge got its ACCEPT rules in a different table from the DROP and every forwarded packet was dropped. The result is a cluster that starts fine from preloaded images but cannot reach anything: Failed to connect to registry.k8s.io port 443: Connection timed out Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen" dial tcp 10.96.0.1:443: i/o timeout DNS still resolves throughout, because the node's resolver is the bridge gateway -- an INPUT path that never touches FORWARD. That asymmetry makes this look like a TLS or registry problem rather than a host firewall one. Pick the legacy backend when `ip_tables` is loaded, which is the signal that something in this namespace already uses it. The nested daemon then writes its ACCEPT rules into the same table as the DROP policy and they take effect before it. This is what the upstream docker-in-docker devcontainer feature does; we cannot use that feature because Docker is installed directly in the image. On a host using nft the condition does not match and nothing changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Moves essential dev-container startup tasks to postStartCommand, supporting SSH-only Codespaces sessions and nested Docker networking.
Changes:
- Adds cache-permission repair and Docker startup scripts.
- Aligns Docker’s iptables backend with the Codespaces host.
- Runs startup prerequisites before attach-time setup.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
.devcontainer/devcontainer.json |
Runs startup scripts on every container start. |
.devcontainer/fix-permissions.sh |
Repairs cache-volume ownership. |
.devcontainer/start-docker.sh |
Configures iptables and starts Docker. |
.devcontainer/setup.sh |
Delegates prerequisites to the new scripts. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+42
to
+46
| # Skip the recursive chown when the top-level owner is already correct -- | ||
| # these trees hold tens of thousands of files and this runs on every start. | ||
| if [ "$(stat -c '%U' "$dir")" != "$USER_NAME" ]; then | ||
| echo "🔑 Fixing ownership of $dir..." | ||
| sudo chown -R "$USER_NAME:$USER_NAME" "$dir" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the dev container usable over
gh codespace ssh, and fixes the nested Docker daemon having no working network on Codespaces.Why
postAttachCommandnever fires for SSH sessionssetup.shis wired topostAttachCommand, which only runs when an editor client attaches. Connecting withgh codespace sshskipped it entirely, so the container ended up with no Docker daemon and root-owned cache volumes. This surfaced as four unrelated-looking failures:Cannot connect to the Docker daemonopen /go/pkg/sumdb/sum.golang.org/latest: no such file or directoryerror Error: EACCES: permission denied, mkdir '/home/codespace/.yarn/cache/v6'The nested Docker daemon had no outbound network
Codespaces runs the dev container in the host network namespace, so the Codespaces VM's own Docker daemon and the daemon we start here write firewall rules into the same namespace. The outer daemon is an older release on the iptables-legacy backend and sets
FORWARD policy DROP, opening holes only for its owndocker0. Our daemon defaulted to the nft backend, so minikube's bridge got its ACCEPT/MASQUERADE rules in a table that does not contain the DROP, and every forwarded packet died at the legacy policy.Only forwarded traffic was affected — DNS kept working because the node's resolver is the bridge gateway, i.e. the host itself, which takes the INPUT path and never traverses FORWARD. That asymmetry made it look like a TLS or registry problem;
curl https://1.1.1.1/from inside the node (no DNS, no cert, no registry) timing out while the host got a 200 is what identified it as the host firewall.How
setup.shinto two single-purpose scripts and run them frompostStartCommand, which fires on every container start regardless of how the user connects:fix-permissions.sh— chowns the mounted cache volumes to the container user (skips the recursive chown when ownership is already correct)start-docker.sh— aligns the iptables backend with the host daemon, then starts dockerd viaservice docker startso the log survives at/var/log/docker.logip_tablesis loaded, i.e. when something in this namespace already uses the legacy backend. On an nft host the block is a no-op. This mirrors what the upstreamdocker-in-dockerdevcontainer feature does; we cannot use that feature because Docker is installed directly in the image.setup.shstill calls both (an editor attach repairs anything that drifted mid-session) but no longer owns the logic. Moving the calls to the top ofmain()also fixes a pre-existing ordering bug wherecleanup_docker_if_neededshelled out todockerbefore dockerd was started.Testing
Verified on a Codespace reached over
gh codespace ssh:make -C tools/dev start-minikube— the ingress addon now pulls its images (previouslyImagePullBackOff)make setup-localenvandmake deploy-bucketeer—STATUS: deployedNotes / out of scope
vendor/is gitignored and only generated bysetup.sh, so SSH-only sessions still need a manualmake vendorbefore building.tools/dev/Makefilestarts minikube with--memory max, which leaves nothing for the host.