Skip to content

Commit 5774ba9

Browse files
authored
feat(upstream): honor Retry-After on 429 via transport RoundTripper (#1042)
* feat(upstream): honor Retry-After on 429 via transport RoundTripper mcp-go flattens every non-2xx upstream response into an error string, so a 429's Retry-After header was gone before any connection-state code ran and the reconnect ladder paced a rate-limited upstream on its own schedule. Capture the hint where the *http.Response still exists: a RetryAfterTransport RoundTripper installed under every HTTP/SSE client we hand to mcp-go, feeding a per-server RetryAfterRecorder. The managed client stamps the resulting deadline onto the state machine, and ConnectionInfo.ShouldAutoReconnect / ShouldRetry treat it as a floor under the existing backoff — effective delay is max(backoff, Retry-After), capped at one hour. - Both RFC 7231 forms parse (delta-seconds and HTTP-date); a malformed, absent, or already-elapsed value falls back to the existing ladder untouched. - 503 counts only when it carries a parseable hint; a bare 503 is an outage the ladder already paces. - The wrapper composes with the existing tracing transport and sits below mcp-go's OAuth handling, so the three authorize paths are unchanged. The two OAuth client constructors now receive our basic client explicitly, since OAuthConfig.HTTPClient covers only metadata/DCR/token requests. - The window is cleared on a successful connect and on Reset, so a stale hint cannot hold back a later, unrelated reconnect. Closes #1040. Refs #1013, #1039. * fix(upstream): scope the Retry-After window to its attempt and gate the OAuth ladder Cross-model review of #1042 found four real holes: - The hint only reached the state machine after a failed Connect, so a 429 answered to the health-check ping was recorded and then ignored. syncRetryAfterFromTransport is now called from every path that turns an upstream failure into Error state. - The recorder was cleared only on success, so a window from an earlier attempt could be re-attributed to a later, unrelated failure — re-parking a server the upstream had not rate-limited again (notably right after a manual reconnect reset the state machine). Each attempt now starts with a clean slate. - ShouldRetryOAuth (the monitoring loop's reconnect path) ignored the window, and its rungs can be shorter than what the upstream asked for. - Converting delta-seconds to a Duration before capping overflowed int64 nanoseconds past ~292 years, turning a huge header into a tiny or negative delay instead of the cap. The bound check now precedes the multiplication. Tests: OAuth streamable-HTTP and SSE clients now have wire-level coverage (a pre-seeded token store makes the request actually go out, so the 429 is observable) — verified to fail when either constructor loses the basic-client option; plus per-attempt window scoping at the core level, the OAuth-ladder gate, and the two overflow-shaped Retry-After inputs. Refs #1040. * fix(upstream): make the 429 sync reachable from the request paths and swap recorders per attempt Round 2 of the cross-model review found the round-1 health-path fix was dead code and the per-attempt clear was too blunt: - performHealthCheck only reached the sync after isConnectionError, which matches transport failures ("connection refused", "deadline exceeded") and not HTTP statuses — so a ping answered 429 never handed its window to the state machine. The sync now runs on any failed ping, before classification; it does not change the health verdict, it only stops the automatic reconnect paths from returning early. The same sync now also runs on a failed tools/call, the other surface that can be rate-limited with no connect attempt in sight. - Clearing one shared recorder at the start of an attempt let a request still in flight on a superseded transport write into the new attempt's slate (and discarded a hint recorded moments before the clear). The core client now holds a generation pointer and swaps in a fresh recorder per attempt, so each transport keeps the recorder it was built with. Tests: the health-path test is verified to fail when the sync moves back behind isConnectionError, plus tools/call coverage. Refs #1040.
1 parent ca00071 commit 5774ba9

14 files changed

Lines changed: 1311 additions & 95 deletions
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
package supervisor
2+
3+
import (
4+
"testing"
5+
"time"
6+
7+
"github.com/stretchr/testify/require"
8+
"go.uber.org/zap"
9+
10+
"github.com/smart-mcp-proxy/mcpproxy-go/internal/config"
11+
"github.com/smart-mcp-proxy/mcpproxy-go/internal/runtime/configsvc"
12+
"github.com/smart-mcp-proxy/mcpproxy-go/internal/upstream/types"
13+
)
14+
15+
// TestSupervisor_Reconcile_RespectsRetryAfter is the supervisor half of #1040's
16+
// acceptance: while a rate-limited upstream's Retry-After window is open the
17+
// periodic reconciliation must plan no ActionConnect, even though the ordinary
18+
// exponential-backoff window has long since elapsed.
19+
func TestSupervisor_Reconcile_RespectsRetryAfter(t *testing.T) {
20+
cfg := &config.Config{
21+
Listen: "127.0.0.1:8080",
22+
Servers: []*config.ServerConfig{
23+
{Name: "rate-limited-server", Enabled: true},
24+
},
25+
}
26+
27+
configSvc := configsvc.NewService(cfg, "/tmp/config.json", zap.NewNop())
28+
defer configSvc.Close()
29+
30+
mockUpstream := NewMockUpstreamAdapter()
31+
defer mockUpstream.Close()
32+
33+
supervisor := New(configSvc, mockUpstream, zap.NewNop())
34+
35+
require.NoError(t, supervisor.reconcile(configSvc.Current()))
36+
supervisor.actionWg.Wait()
37+
38+
setConnectionState := func(info *types.ConnectionInfo) {
39+
mockUpstream.mu.Lock()
40+
defer mockUpstream.mu.Unlock()
41+
mockUpstream.connected["rate-limited-server"] = false
42+
if state, ok := mockUpstream.states["rate-limited-server"]; ok {
43+
state.Connected = false
44+
state.ConnectionInfo = info
45+
}
46+
}
47+
isConnected := func() bool {
48+
mockUpstream.mu.Lock()
49+
defer mockUpstream.mu.Unlock()
50+
return mockUpstream.connected["rate-limited-server"]
51+
}
52+
reconcileAndDrain := func() {
53+
t.Helper()
54+
require.NoError(t, supervisor.reconcile(configSvc.Current()))
55+
supervisor.actionWg.Wait()
56+
}
57+
58+
// `Retry-After: 3600` on a 429: the ladder says "dial now" (its window
59+
// elapsed long ago), the upstream says "not for an hour". The upstream wins.
60+
setConnectionState(&types.ConnectionInfo{
61+
State: types.StateError,
62+
RetryCount: 1,
63+
LastRetryTime: time.Now().Add(-time.Hour),
64+
RetryAfter: time.Now().Add(time.Hour),
65+
})
66+
reconcileAndDrain()
67+
require.False(t, isConnected(), "supervisor re-dialed a server inside its Retry-After window")
68+
69+
// Once the window has passed the normal ladder takes over again.
70+
setConnectionState(&types.ConnectionInfo{
71+
State: types.StateError,
72+
RetryCount: 1,
73+
LastRetryTime: time.Now().Add(-time.Hour),
74+
RetryAfter: time.Now().Add(-time.Second),
75+
})
76+
reconcileAndDrain()
77+
require.True(t, isConnected(), "supervisor never came back after the Retry-After window elapsed")
78+
}

‎internal/transport/http.go‎

Lines changed: 97 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,36 @@ type HTTPTransportConfig struct {
137137
// credential broker can drive injection without this package importing it.
138138
BrokeredAuth *BrokeredAuth
139139
TraceEnabled bool // Enable detailed HTTP/SSE frame tracing
140+
// RetryAfter, when set, receives the `Retry-After` hints observed on this
141+
// upstream's rate-limited responses (#1040). mcp-go flattens non-2xx
142+
// responses into strings, so a RoundTripper is the last place the header
143+
// still exists. nil disables the wrapper entirely (unchanged behaviour).
144+
RetryAfter *RetryAfterRecorder
145+
}
146+
147+
// upstreamRoundTripper composes the outbound transport wrappers for an upstream
148+
// HTTP/SSE client: frame tracing (opt-in) and the Retry-After recorder (#1040).
149+
// It always WRAPS the given base rather than replacing it, and it sits strictly
150+
// below mcp-go's OAuth handling — the OAuth authorize/token wrappers live above
151+
// this layer and are untouched.
152+
func (cfg *HTTPTransportConfig) upstreamRoundTripper(base http.RoundTripper, logger *zap.Logger) http.RoundTripper {
153+
rt := base
154+
if rt == nil {
155+
rt = http.DefaultTransport
156+
}
157+
if cfg.TraceEnabled {
158+
rt = NewLoggingTransport(rt, logger)
159+
}
160+
if cfg.RetryAfter != nil {
161+
rt = NewRetryAfterTransport(rt, cfg.RetryAfter, logger)
162+
}
163+
return rt
164+
}
165+
166+
// needsCustomTransport reports whether this config requires us to hand mcp-go
167+
// our own *http.Client instead of letting it build the default one.
168+
func (cfg *HTTPTransportConfig) needsCustomTransport() bool {
169+
return cfg.TraceEnabled || cfg.RetryAfter != nil
140170
}
141171

142172
// effectiveHeaders returns the outbound header set, applying brokered per-user
@@ -202,7 +232,20 @@ func CreateHTTPClient(cfg *HTTPTransportConfig) (*client.Client, error) {
202232
zap.String("note", "Extra parameters will be injected into OAuth requests"))
203233
}
204234

205-
client, err := client.NewOAuthStreamableHttpClient(cfg.URL, *cfg.OAuthConfig)
235+
// mcp-go builds the OAuth client's transport itself; OAuthConfig.HTTPClient
236+
// only covers the OAuth metadata/DCR/token requests, NOT the MCP requests.
237+
// Passing our own basic client is therefore the only way the Retry-After
238+
// recorder (#1040) sees a 429 on the MCP endpoint. No Timeout is set, to
239+
// preserve mcp-go's default for this branch (OAuth flows rely on the
240+
// caller's context deadline, which can be up to 30 minutes).
241+
var oauthOpts []transport.StreamableHTTPCOption
242+
if cfg.needsCustomTransport() {
243+
oauthOpts = append(oauthOpts, transport.WithHTTPBasicClient(&http.Client{
244+
Transport: cfg.upstreamRoundTripper(http.DefaultTransport, logger),
245+
}))
246+
}
247+
248+
client, err := client.NewOAuthStreamableHttpClient(cfg.URL, *cfg.OAuthConfig, oauthOpts...)
206249
if err != nil {
207250
logger.Error("Failed to create OAuth client", zap.Error(err))
208251
return nil, fmt.Errorf("failed to create OAuth client: %w", err)
@@ -219,49 +262,40 @@ func CreateHTTPClient(cfg *HTTPTransportConfig) (*client.Client, error) {
219262
// inbound gateway/IdP token (FR-017).
220263
headers := cfg.effectiveHeaders()
221264

222-
// If tracing is enabled, create HTTP client with logging transport
223-
if cfg.TraceEnabled {
224-
logger.Info("🔍 HTTP TRACE MODE ENABLED - All HTTP traffic will be logged")
225-
baseTransport := &http.Transport{
226-
MaxIdleConns: 10,
227-
IdleConnTimeout: 90 * time.Second,
228-
ResponseHeaderTimeout: 30 * time.Second,
229-
}
230-
loggingTransport := NewLoggingTransport(baseTransport, logger)
231-
httpClient := &http.Client{
232-
Transport: loggingTransport,
233-
Timeout: 180 * time.Second,
234-
}
235-
236-
var httpTransport transport.Interface
237-
var err error
238-
if len(headers) > 0 {
239-
httpTransport, err = transport.NewStreamableHTTP(cfg.URL,
240-
transport.WithHTTPHeaders(headers),
241-
transport.WithHTTPBasicClient(httpClient))
242-
} else {
243-
httpTransport, err = transport.NewStreamableHTTP(cfg.URL,
244-
transport.WithHTTPBasicClient(httpClient))
245-
}
246-
if err != nil {
247-
return nil, fmt.Errorf("failed to create HTTP transport with tracing: %w", err)
248-
}
249-
return client.NewClient(httpTransport), nil
250-
}
251-
252-
// Use regular HTTP client
265+
opts := []transport.StreamableHTTPCOption{}
253266
if len(headers) > 0 {
254267
logger.Debug("Adding HTTP headers", zap.Int("header_count", len(headers)))
255-
httpTransport, err := transport.NewStreamableHTTP(cfg.URL,
256-
transport.WithHTTPHeaders(headers))
257-
if err != nil {
258-
return nil, fmt.Errorf("failed to create HTTP transport: %w", err)
268+
opts = append(opts, transport.WithHTTPHeaders(headers))
269+
}
270+
271+
switch {
272+
case cfg.needsCustomTransport():
273+
// Timeout policy is preserved exactly as it was before #1040: the trace
274+
// client and the header-less client carry a 180s request timeout, while
275+
// the headers variant has always run on mcp-go's default (no timeout),
276+
// which long-running tool calls depend on.
277+
base := http.RoundTripper(http.DefaultTransport)
278+
timeout := time.Duration(0)
279+
if cfg.TraceEnabled {
280+
logger.Info("🔍 HTTP TRACE MODE ENABLED - All HTTP traffic will be logged")
281+
base = &http.Transport{
282+
MaxIdleConns: 10,
283+
IdleConnTimeout: 90 * time.Second,
284+
ResponseHeaderTimeout: 30 * time.Second,
285+
}
286+
timeout = 180 * time.Second
287+
} else if len(headers) == 0 {
288+
timeout = 180 * time.Second
259289
}
260-
return client.NewClient(httpTransport), nil
290+
opts = append(opts, transport.WithHTTPBasicClient(&http.Client{
291+
Transport: cfg.upstreamRoundTripper(base, logger),
292+
Timeout: timeout,
293+
}))
294+
case len(headers) == 0:
295+
opts = append(opts, transport.WithHTTPTimeout(180*time.Second)) // Increased timeout for HTTP connections
261296
}
262297

263-
httpTransport, err := transport.NewStreamableHTTP(cfg.URL,
264-
transport.WithHTTPTimeout(180*time.Second)) // Increased timeout for HTTP connections
298+
httpTransport, err := transport.NewStreamableHTTP(cfg.URL, opts...)
265299
if err != nil {
266300
return nil, fmt.Errorf("failed to create HTTP transport: %w", err)
267301
}
@@ -313,7 +347,18 @@ func CreateSSEClient(cfg *HTTPTransportConfig) (*client.Client, error) {
313347
zap.String("note", "Extra parameters will be injected into OAuth requests"))
314348
}
315349

316-
client, err := client.NewOAuthSSEClient(cfg.URL, *cfg.OAuthConfig)
350+
// As in the streamable-HTTP OAuth branch: OAuthConfig.HTTPClient covers
351+
// only the OAuth metadata/DCR/token calls, so the MCP requests need our
352+
// own client for the Retry-After recorder to see a 429 (#1040). No
353+
// Timeout — SSE streams are long-lived by design.
354+
var oauthOpts []transport.ClientOption
355+
if cfg.needsCustomTransport() {
356+
oauthOpts = append(oauthOpts, client.WithHTTPClient(&http.Client{
357+
Transport: cfg.upstreamRoundTripper(http.DefaultTransport, logger),
358+
}))
359+
}
360+
361+
client, err := client.NewOAuthSSEClient(cfg.URL, *cfg.OAuthConfig, oauthOpts...)
317362
if err != nil {
318363
logger.Error("Failed to create OAuth SSE client", zap.Error(err))
319364
return nil, fmt.Errorf("failed to create OAuth SSE client: %w", err)
@@ -330,48 +375,6 @@ func CreateSSEClient(cfg *HTTPTransportConfig) (*client.Client, error) {
330375
// inbound gateway/IdP token (FR-017).
331376
headers := cfg.effectiveHeaders()
332377

333-
// Use regular SSE client
334-
if len(headers) > 0 {
335-
logger.Debug("Adding SSE headers", zap.Int("header_count", len(headers)))
336-
// Create custom HTTP client for SSE - NO Timeout field to allow indefinite streaming
337-
// The Timeout field covers the entire request duration, which kills long-lived SSE streams
338-
// Instead, we rely on IdleConnTimeout to detect dead connections
339-
baseTransport := &http.Transport{
340-
MaxIdleConns: 10,
341-
IdleConnTimeout: 300 * time.Second, // 5 minutes idle before closing
342-
DisableCompression: false,
343-
DisableKeepAlives: false, // Enable keep-alives for SSE stability
344-
MaxIdleConnsPerHost: 5,
345-
// ResponseHeaderTimeout can be used to timeout initial connection, but not ongoing stream
346-
ResponseHeaderTimeout: 30 * time.Second,
347-
}
348-
349-
var transport http.RoundTripper = baseTransport
350-
if cfg.TraceEnabled {
351-
logger.Info("🔍 SSE TRACE MODE ENABLED - All HTTP traffic and SSE frames will be logged")
352-
transport = NewLoggingTransport(baseTransport, logger)
353-
}
354-
355-
httpClient := &http.Client{
356-
Transport: transport,
357-
}
358-
359-
logger.Info("Creating SSE MCP client with indefinite timeout for long-lived streams",
360-
zap.String("url", cfg.URL),
361-
zap.Duration("idle_timeout", 300*time.Second),
362-
zap.Duration("header_timeout", 30*time.Second),
363-
zap.Int("header_count", len(headers)),
364-
zap.String("note", "Removed http.Client.Timeout to allow SSE streams longer than 3 minutes"))
365-
366-
sseClient, err := client.NewSSEMCPClient(cfg.URL,
367-
client.WithHTTPClient(httpClient),
368-
client.WithHeaders(headers))
369-
if err != nil {
370-
return nil, fmt.Errorf("failed to create SSE client: %w", err)
371-
}
372-
return sseClient, nil
373-
}
374-
375378
// Create custom HTTP client for SSE - NO Timeout field to allow indefinite streaming
376379
// The Timeout field covers the entire request duration, which kills long-lived SSE streams
377380
// Instead, we rely on IdleConnTimeout to detect dead connections
@@ -385,20 +388,22 @@ func CreateSSEClient(cfg *HTTPTransportConfig) (*client.Client, error) {
385388
ResponseHeaderTimeout: 30 * time.Second,
386389
}
387390

388-
var transport http.RoundTripper = baseTransport
389391
if cfg.TraceEnabled {
390392
logger.Info("🔍 SSE TRACE MODE ENABLED - All HTTP traffic and SSE frames will be logged")
391-
transport = NewLoggingTransport(baseTransport, logger)
392393
}
393394

395+
// Tracing and the Retry-After recorder (#1040) both wrap the base transport;
396+
// the SSE-specific tuning above and the deliberately absent Client.Timeout
397+
// are unchanged.
394398
httpClient := &http.Client{
395-
Transport: transport,
399+
Transport: cfg.upstreamRoundTripper(baseTransport, logger),
396400
}
397401

398402
logger.Info("Creating SSE MCP client with indefinite timeout for long-lived streams",
399403
zap.String("url", cfg.URL),
400404
zap.Duration("idle_timeout", 300*time.Second),
401405
zap.Duration("header_timeout", 30*time.Second),
406+
zap.Int("header_count", len(headers)),
402407
zap.String("note", "Removed http.Client.Timeout to allow SSE streams longer than 3 minutes"))
403408

404409
// Enhanced trace-level debugging for SSE transport
@@ -411,8 +416,13 @@ func CreateSSEClient(cfg *HTTPTransportConfig) (*client.Client, error) {
411416
zap.String("debug_note", "SSE client will establish persistent connection for JSON-RPC over SSE with no overall timeout"))
412417
}
413418

414-
sseClient, err := client.NewSSEMCPClient(cfg.URL,
415-
client.WithHTTPClient(httpClient))
419+
sseOpts := []transport.ClientOption{client.WithHTTPClient(httpClient)}
420+
if len(headers) > 0 {
421+
logger.Debug("Adding SSE headers", zap.Int("header_count", len(headers)))
422+
sseOpts = append(sseOpts, client.WithHeaders(headers))
423+
}
424+
425+
sseClient, err := client.NewSSEMCPClient(cfg.URL, sseOpts...)
416426
if err != nil {
417427
return nil, fmt.Errorf("failed to create SSE client: %w", err)
418428
}

0 commit comments

Comments
 (0)