[Metrics] Measure the SSH round trip past the API server

[zpoint]

Problem

sky_apiserver_websocket_ssh_latency_seconds is widely read as keystroke latency. It is neither keystroke latency nor a measurement of the path to the pod:

• It is a synthetic PING the client's ws-proxy sends every 10s (websocket_proxy.py, HEARTBEAT_INTERVAL_SECONDS), not something a keypress triggers.
• The server echoes that PING and continues without forwarding it (websocket_utils.py), so the kubectl port-forward tunnel and the pod's sshd are excluded by construction.
• Because the PONG cannot be sent until the websocket read loop is free, it doubles as an event-loop responsiveness probe.

The practical consequence: a short interactive session can produce zero samples while every keystroke is measurably slow. Measured on a kind cluster, a ~12s ssh session recorded ..._count = 0 on the heartbeat while 29 keystroke round trips were happening.

What this adds

sky_apiserver_ssh_backend_turnaround_seconds{path} covers exactly the excluded leg: API server → port-forward → kubelet → pod sshd → shell echo → back.

It adds no bytes to the wire and needs no client support. The proxy loop already unpacks a header per frame, so a keystroke-sized write to the backend is timestamped and paired with the next read. A stream of keystrokes is an ordered request/response conversation, which is what makes the pairing sound; when typing pipelines, the oldest outstanding stamp is kept, since the first read answers the first write. Samples are dropped when the write is too large to be a keystroke (a paste also clears any outstanding stamp) or when the reply takes longer than 2s and is more likely unrelated output than a very slow echo.

This is a distribution, not a per-keystroke truth — backend traffic that answers no write can attach a read to the wrong write. The docstring says so.

sky_apiserver_ssh_sessions_total{path} is the denominator. Without it an empty histogram is ambiguous: nobody is SSHing, every session was redirected away by a plugin hook, or the pairing broke. The redirect branch increments it before returning, so a handed-off session is still counted.

Also here:

• The heartbeat metric's Help string now says what it is, what it excludes, and when it is empty rather than zero.
• The "SSH RTT to API Server P95" dashboard panel is retitled accordingly, with panels added for the two new metrics.
• time.monotonic() replaces time.time() for the heartbeat interval. Both ends are read in one process, so the wall clock buys nothing and costs correctness: an NTP step or a laptop suspend inside the 10s window is charged to SSH latency, and a backwards step yields a negative interval that wraps when packed into the unsigned !Q.
• The new metrics get a bucket ladder starting at 1ms. _LATENCY_BUCKETS starts at 5ms because it is sized for request durations; this leg measures a ~3ms median in-cluster, so every healthy value would land in one bucket.

Testing

15 unit tests in tests/unit_tests/test_sky/server/test_websocket_utils.py cover the pairing rules and drive the real run_websocket_proxy loop, including that a heartbeat PING must not produce a turnaround sample.

Measured end to end against a kind cluster, typing over a real ssh:

• 284 paired samples at a ~3ms median, ~76% of keystrokes sampled.
• Pausing the node container for 0.7s mid-keystroke put exactly three samples in the (0.5s, 1s] bucket, while the heartbeat — 17 pings in the same window — stayed at a 0.88ms mean, blind to the stall.
• Verified in the deployed process shape (--deploy, 10 server processes): samples written by a worker process are correctly aggregated by the multiprocess collector into the /metrics served from the main process.
• promtool check metrics on the live endpoint reports no issues for the new metrics.

An earlier version of the pairing discarded a sample whenever a second keystroke arrived before the first was answered. The end-to-end stall test showed that throws away precisely the samples worth having, since a stalled backend is exactly when a user keeps typing into the lag. Hence keeping the oldest stamp.

---

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🟡 SSH panels obscure internal metrics

The new panels reuse gridPos rows occupied by the Internals section. Grafana overlays unrelated charts, making both sections unreadable.

Prompt for agents

Resolve the Grafana grid collision in charts/skypilot/manifests/api-server-overview.json. The new SSH backend panel starts at y=47 and the sessions panel at y=55, while the existing full-width Internals row starts at y=47 and its panels occupy the same ranges. Reposition the Internals row and every panel beneath it far enough downward to preserve the two new eight-row SSH panels, then verify that no gridPos rectangles overlap.

---

Was this helpful? React with 👍 or 👎 to provide feedback.