Skip to content

Add internal sandbox WebSocket SSH server - #7588

Merged
HsingTang merged 7 commits into
mainfrom
HsingTang/stack/sandbox-websocket-server
Nov 19, 2026
Merged

HsingTang merged 7 commits into
mainfrom
HsingTang/stack/sandbox-websocket-server

Conversation

@HsingTang

@HsingTang HsingTang commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR

Use this link to review incremental changes.


What did you change, and why?

Sandbox SSH needs a server that works through HTTP/WebSocket port forwarding without compute initialization. Add the hidden, internal databricks sandbox server command while leaving the existing SSH command unchanged in this PR.

Keep the implementation under experimental/ssh/sandbox with sandbox CODEOWNERS. Require an invocation token, validate forwarding ports, and allow one SSH session with the existing handover and versioned resume protocols. The shared proxy owns session admission and reports completion or process-start failure; invoking the process factory consumes the single session, while failed WebSocket upgrades can retry. The sandbox owns its first-connection deadline and shutdown. Preserve compute timeout defaults and remove unused proxy options.

How do you know it works?

Sandbox and compute SSH suites, race tests, acceptance tests, and ./task fmt checks lint pass. Tests cover repeated WebSocket interruptions without losing bytes or restarting the SSH process, handover through a proxy with a connection lifetime limit, failed upgrades and process starts, startup deadlines, and cleanup. The SIGTERM acceptance test verifies a child that remains alive when stdin closes. A standalone native OpenSSH check built from this PR verifies that both SSH server processes exit when the server receives SIGTERM.

The dependent client was also validated against a real sandbox, including SSH, SCP/SFTP, and a 28-minute session with forced-disconnect recovery and scheduled handover. Local Isaac review findings were addressed or checked against native OpenSSH behavior. The latest startup-failure correction passed repeated race tests with 16 concurrent dials, the full SSH suite, and a fresh local Isaac review. Its dependent client also passed fresh real-sandbox SSH/SCP/SFTP, stopped-sandbox startup, eight-second outage recovery, and failure-log retention checks.

Validation after nonblocking review fixes (2026-11-19)

At cd5c2165022ec908f438fdf9085d2b5a3636f0a0, the PID stub publishes its completed file using a same-directory rename, and the concurrent startup test waits for all HTTP handlers before checking completion counts. The fixed test passed 3,000 repetitions with Go's race detector. The server acceptance script passed 1,000 runs with 16 concurrent workers. Formatting, lint, and local Isaac review passed with no remaining reported findings.

This PR was written with Codex.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: cd5c216

Run: 44631896126

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 20 5:12
💚​ aws-windows-latest-4core-16gb 1 274 18 4:07
💚​ azure linux-2core-8gb 1 271 20 4:56
💚​ azure-windows-latest-4core-16gb 1 273 18 4:38
💚​ gcp linux-2core-8gb 1 272 20 6:08
💚​ gcp-windows-latest-4core-16gb 1 274 18 4:13
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:35 azure-windows-latest-4core-16gb TestAccept
4:12 gcp-windows-latest-4core-16gb TestAccept
4:06 aws-windows-latest-4core-16gb TestAccept

@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-websocket-server branch from d3e8539 to e27bc56 Compare November 13, 2026 21:16
@HsingTang HsingTang changed the title Add sandbox WebSocket SSH server Add internal sandbox WebSocket SSH server Nov 13, 2026
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-websocket-server branch from e27bc56 to 36dc63c Compare November 13, 2026 22:12
@HsingTang
HsingTang marked this pull request as ready for review November 13, 2026 22:18
@HsingTang
HsingTang requested review from a team as code owners November 13, 2026 22:18
@rclarey
rclarey requested review from rclarey and removed request for rugpanov November 16, 2026 18:23

@rclarey rclarey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The relay looks careful overall. Admission and the idle decision share a lock, readiness is announced only after sshd -t passes and the listener is bound, ReadHeaderTimeout is set, and shutdown waits for in-flight handlers. The ConnectionsManager change also looks safe for compute, because waitForIdleShutdown rechecks Count(). I ran the Server tests 5 times with -race and they all passed.

Inline comments cover the specific issues. Two broader points, compared with ssh connect:

  • Resume and handover. These are the main reliability features of the compute tunnel, and this relay rejects both. The README in #7590 acknowledges that a broken connection ends the session. Does sandbox port forwarding enforce a maximum WebSocket lifetime, like the driver proxy? If it does, long sessions will drop on a schedule. Handover exists in compute to work around exactly that.
  • Logging. The compute server runs with --log-level=info --log-file=stdout and logs lifecycle events such as idle shutdown. Serve logs nothing: not readiness, not 503 rejections, not idle shutdown. #7589 doesn't pass a log level either, so only warnings reach the client's diagnostics stream. That leaves nothing to look at when a connection fails to come up. It would help to log the lifecycle events at info level and have #7589 pass --log-level.

Comment thread cmd/sandbox/server.go Resolved
Comment thread experimental/ssh/sandbox/server.go Resolved
Comment thread experimental/ssh/sandbox/server.go Resolved
Comment thread cmd/sandbox/server.go Resolved
Comment thread experimental/ssh/sandbox/server.go Resolved
Comment thread experimental/ssh/sandbox/server.go Resolved
Comment thread cmd/sandbox/server_test.go Resolved
Comment thread .golangci.yaml Resolved

@anton-107 anton-107 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for code ownership of experimental/ssh/sandbox/, see the inline comment. It's the CODEOWNERS side of rclarey's depguard comment on .golangci.yaml:60.

Comment thread experimental/ssh/sandbox/server.go Resolved
Comment thread experimental/ssh/sandbox/server.go Resolved
Comment thread experimental/ssh/sandbox/server.go Resolved
Comment thread cmd/sandbox/server.go Resolved
Comment thread cmd/sandbox/server.go Resolved
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-websocket-server branch from 36dc63c to cb5c0cf Compare November 18, 2026 05:45
@HsingTang

Copy link
Copy Markdown
Contributor Author

@rclarey Addressed the two broader review points as well. The sandbox client now renews the WebSocket every 25 minutes using same-session handover; unexpected-disconnect resume remains disabled. A local reverse-proxy test enforces a short WebSocket lifetime and verifies that repeated handovers keep the same SSH process and data stream alive.

The server now logs readiness, rejected connections, first-connection timeout, and shutdown at info level. The client starts it with --log-level info and sends diagnostics to stderr, keeping stdout available for SSH traffic. These changes are in cb5c0cf and 637d42e. Local tests and linters pass; live workspace validation is still pending.

@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-websocket-server branch from cb5c0cf to a6b20fd Compare November 18, 2026 06:01
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Commit: cd5c21650 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (06e25a1) latest (v1.27.0)
TestAccept/cmd/sandbox/server ✅ ❌ ❌
TestAccept/cmd/sandbox/server: branch ✅ | main (06e25a1) ❌ | latest (v1.27.0) ❌

main (06e25a1):

acceptance_test.go:1400: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1400
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1141
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:686
        	Error:      	Not equal: 
        	            	expected: "\n>>> python3 server_test.py\nServer remains hidden from sandbox help\nServer starts without workspace credentials\nSIGTERM stops the server and its SSH child\n"
        	            	actual  : "\n>>> python3 server_test.py\nServer remains hidden from sandbox help\nTraceback (most recent call last):\n  File \"[TEST_TMP_DIR]/server_test.py\", line 46, in <module>\n    ready = json.loads(process.stdout.readline())\n  File \"/home/runner/work/_temp/uv-python-dir/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/json/__init__.py\", line 346, in loads\n    return _default_decoder.decode(s)\n           ~~~~~~~~~~~~~~~~~~~~~~~^^^\n  File \"/home/runner/work/_temp/uv-python-dir/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/json/decoder.py\", line 345, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/runner/work/_temp/uv-python-dir/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/json/decoder.py\", line 363, in raw_decode\n    raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -3,4 +3,16 @@
        	            	 Server remains hidden from sandbox help
        	            	-Server starts without workspace credentials
        	            	-SIGTERM stops the server and its SSH child
        	            	+Traceback (most recent call last):
        	            	+  File "[TEST_TMP_DIR]/server_test.py", line 46, in <module>
        	            	+    ready = json.loads(process.stdout.readline())
        	            	+  File "/home/runner/work/_temp/uv-python-dir/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/json/__init__.py", line 346, in loads
        	            	+    return _default_decoder.decode(s)
        	            	+           ~~~~~~~~~~~~~~~~~~~~~~~^^^
        	            	+  File "/home/runner/work/_temp/uv-python-dir/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/json/decoder.py", line 345, in decode
        	            	+    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
        	            	+               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
        	            	+  File "/home/runner/work/_temp/uv-python-dir/cpython-3.13.6-linux-x86_64-gnu/lib/python3.13/json/decoder.py", line 363, in raw_decode
        	            	+    raise JSONDecodeError("Expecting value", s, err.value) from None
        	            	+json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/cmd/sandbox/server
        	Messages:   	cmd/sandbox/server/output.txt vs /tmp/TestAcceptcmdsandboxserver588643451/002/output.txt
FAIL

latest (v1.27.0): same output as main (06e25a1)

Comment thread experimental/ssh/sandbox/server.go Resolved

@jaecho-db jaecho-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread experimental/ssh/sandbox/server_test.go Resolved
Comment thread acceptance/cmd/sandbox/server/test.toml Resolved
@HsingTang
HsingTang force-pushed the HsingTang/stack/sandbox-websocket-server branch from a6b20fd to 219d06c Compare November 19, 2026 05:15
@HsingTang
HsingTang requested a review from rclarey November 19, 2026 05:23
closing := false
handler := proxy.NewProxyServer(ctx, connections, func(ctx context.Context) *exec.Cmd {
mu.Lock()
close(started)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a failed sshd start the proxy frees the session for a retry, but this factory closes started unconditionally, so a second dial that lands before Serve sets closing panics here with mu held, and Serve can then hang in its deferred mu.Lock() with SIGTERM already captured by NotifyContext. Our client only dials one new session per server so it's unlikely in practice, but guarding the close with a sync.Once (or keeping the session consumed on a start failure, since Serve exits on it anyway) would close it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by keeping single-session admission consumed once the process factory is invoked. Only a failed WebSocket upgrade releases it. An sshd startup error still reaches Serve, but another request cannot invoke the factory and close started again. Tests cover stdin/stdout/Start failures, rejection of a second session, and 16 concurrent startup attempts; repeated race checks pass.

@anton-107 anton-107 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for one issue, see the inline comment on server.go:64. It's the double close from rclarey's round-2 comment, which I reproduced. Everything else from the earlier rounds looks addressed, and the CODEOWNERS entry resolves my previous request.

closing := false
handler := proxy.NewProxyServer(ctx, connections, func(ctx context.Context) *exec.Cmd {
mu.Lock()
close(started)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this panics on a retry after a failed sshd start. It's the same issue as rclarey's round-2 comment above, which is still open.

WithSingleSession sets sessionUsed back to false when the process didn't start, so a second new-id dial can reach this factory again before Serve's deferred closing = true runs. The second close(started) then panics with mu held. net/http recovers the panic, but mu stays locked. If that happens before Serve's defer calls mu.Lock(), Serve deadlocks, and SIGTERM has already been consumed by NotifyContext.

Repro: a fake sshd that passes -t and then deletes itself (rm -f "$0"), so sshd -i fails Start with ENOENT. Then 16 goroutines keep dialing /ssh?id=s<n> with the token. In 6 of 20 runs the log shows http: panic serving ...: close of closed channel. I didn't catch the deadlock itself; in those runs the panic came after the defer had released mu.

The client dials once, so this is unlikely in practice, but the fix is small. I'd keep the session consumed once the factory has run, instead of reopening it in handleNewConnection, because Serve exits on a start failure anyway. That also removes the processStarted reset branch. A sync.Once around the close would work too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied the admission change you suggested: invoking the command factory consumes the single session even if sshd fails to start, while failed WebSocket upgrades remain retryable. Removed the processStarted reset logic. The proxy regression asserts exactly one factory invocation and completion under 16 concurrent dials; the sandbox test also makes 16 concurrent attempts after removing sshd and verifies that Serve returns. Race tests and the full SSH suite pass.

@anton-107 anton-107 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The double close on server.go:64 is fixed in 605f0a0. Once the process factory runs, the session stays used, so a second dial can't reach close(started) again. I reran the repro (fake sshd that deletes itself, 16 goroutines dialing in a loop) and got 0 panics in 40 runs, down from 6 in 20. The sandbox and proxy packages also pass with -race (count=3). LGTM.

@HsingTang
HsingTang added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 0f23a05 Nov 19, 2026
32 checks passed
@HsingTang
HsingTang deleted the HsingTang/stack/sandbox-websocket-server branch November 19, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants