feat(api): add connection-aware readiness probe for SSE - #2787
Merged
Conversation
When SSE connections reach the configured threshold (default 95% of --sse-max-connections), the readiness probe returns 503. This removes the pod from Kubernetes Endpoints and GCLB NEG, preventing new connections from being routed to saturated pods while keeping existing SSE connections alive. The Envoy sidecar circuit breaker is ineffective for SSE over HTTP/2 because multiplexed streams are not counted as active TCP connections or requests. This readiness check bypasses Envoy and reads the Dispatcher's connection count directly. Closes #2786 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds connection-aware SSE readiness checks so saturated API pods stop receiving new traffic without affecting liveness.
Changes:
- Adds synchronous readiness-only health checks.
- Exposes thread-safe SSE connection counts.
- Configures readiness failure at an SSE capacity threshold.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
pkg/health/health.go |
Adds readiness-specific checks. |
pkg/health/health_test.go |
Tests readiness and liveness behavior. |
pkg/api/stream/dispatcher.go |
Exposes current and maximum connections. |
pkg/api/stream/dispatcher_test.go |
Tests connection status tracking. |
pkg/api/cmd/server.go |
Adds and wires the SSE readiness threshold. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Use math.Ceil instead of int truncation for threshold calculation - Validate sse-readiness-threshold is within 0.0-1.0 range - Add duplicate name guard to WithReadinessCheck (same as WithCheck) - Add sseReadinessThreshold to Helm chart values and deployment template Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Return only the current connection count since max is unused by callers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
hvn2k1
reviewed
Aug 26, 2026
hvn2k1
reviewed
Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #2152
What this PR does
Adds a readiness-only health check that returns 503 when SSE connections reach the configured threshold (default 95% of
--sse-max-connections). This removes the pod from Kubernetes Endpoints and GCLB NEG, stopping new connections from being routed to saturated pods while keeping existing SSE connections alive.Background
When a pod hits its SSE connection limit, it returns 503 but the readiness probe continues to return 200. GCLB keeps sending new connections to the saturated pod, causing 503 errors until HPA scale-out catches up. The existing Envoy circuit breaker does not detect SSE connection saturation due to HTTP/2 multiplexing.
This readiness probe reads
Dispatcher.ActiveConns()directly to detect saturation.Verified in dev: 1M SSE load test showed 15x reduction in 5xx rate (867/min vs 13.3K/min baseline).
Points
--sse-readiness-thresholdflag (default: 0.95) controls the fraction at which readiness starts failing--sse-max-connections=16000and threshold 0.95, readiness fails at 15,200 — leaving an 800-connection buffer for probe propagation delay (~3-5s)sseReadinessThresholdadded tovalues.yamlanddeployment.yaml