Respect Retry-After header for BYOK 429 responses

[LittleTOXA]

Feature Description

Add support for the upstream Retry-After header when retrying HTTP 429 responses from BYOK providers.

Some inference providers use rolling/sliding token rate limits and return a Retry-After header indicating the minimum amount of time the client should wait before sending another request.

When Retry-After is present on a 429 response, CommandCode should wait at least for the duration specified by the provider before retrying.

Suggested behavior:

1. Preserve the upstream Retry-After header on BYOK HTTP 429 responses.
2. Parse both standard forms of Retry-After:
   • delay in seconds;
   • HTTP date.
3. Wait at least for the duration specified by Retry-After.
4. If subsequent 429s occur, use exponential backoff with jitter.
5. If Retry-After is absent, fall back to the existing retry/backoff behavior.
6. Avoid immediate or parallel retries while the provider has explicitly requested the client to wait.

This should apply to upstream/BYOK rate limiting and should not change the existing handling of CommandCode's own recognized usage-window limits.

Use Case

I encountered this while using GLM-5.3-Flash through a BYOK provider.

My typical workload consists of long-running coding and reverse-engineering sessions (Ghidra/disassembly). As a session progresses, individual model requests commonly reach 100k–150k+ input tokens.

Baseten enforces its TPM limit per model across the workspace using a rolling/sliding window. Their support confirmed that HTTP 429 responses include a Retry-After header with the minimum time the client should wait before retrying.

During one real CommandCode session, the Baseten dashboard recorded:

• 22 successful responses (HTTP 200)
• 88 rate-limited requests (HTTP 429)

From the CommandCode UI, it appeared that the client was repeatedly reconnecting/retrying while the provider was still rate-limited.

With a 100k–150k token request, retrying too early is particularly inefficient because the provider may still not have enough capacity available in its rolling TPM window for the request.

Respecting Retry-After would allow CommandCode to pause once, resume when the provider indicates capacity should be available, and avoid large numbers of unnecessary retry attempts.

This would also make BYOK integrations behave better with other inference providers that use standard HTTP 429 + Retry-After rate-limit semantics.

Additional Context

I contacted provider's support to clarify the behavior. They confirmed:

The 500k TPM limit is enforced per model across the workspace, rather than per API key, client session, or context.

The limit uses a rolling/sliding window rather than a fixed reset at the top of each minute.

On a 429, the API should return a Retry-After header indicating the minimum number of seconds to wait. We recommend respecting that value and using exponential backoff with jitter rather than retrying at a fixed interval or immediately.

They specifically recommended the following client behavior:

• Read and follow Retry-After when present.
• Apply exponential backoff with jitter for subsequent 429s.
• Avoid parallel or immediate retries.

The high number of 429s was visible directly in the provider dashboard: 22 successful responses vs. 88 HTTP 429 responses during the test period.

How important is this to you?

Not critical, but important

[vipulgupta2048]

Thanks @LittleTOXA for reporting this. We will look into this.