Large non-streaming requests to reasoning models deterministically 504 at ~600s

[Julzilla]

Summary

A large non-streaming request to a reasoning model fails with HTTP 504 Gateway Timeout at ~600 seconds. The user sees a blank result and no error.

stream: False is hardcoded at providers/openai_compatible.py:589. A reasoning model must generate its whole response, reasoning tokens included, before any bytes go out. At ~100k tokens of input that takes longer than the edge proxy's idle timeout, so the proxy cuts the connection.

I am not asking you to switch to streaming. #134 asked for the opposite, and I would rather hand you the evidence than a patch that breaks someone else.

Environment

• Installed via uvx --from git+https://github.com/BeehiveInnovations/pal-mcp-server.git
• Windows 11, Python 3.13
• Custom provider (CUSTOM_API_URL) against an OpenAI-compatible endpoint behind Cloudflare
• A reasoning model that defaults to maximum thinking effort and emits reasoning_content

What happens

codereview inlines file contents into one prompt and makes a single buffered request. From logs/mcp_server.log:

22:51:42  POST /chat/completions          (Expert analysis embedding: 6 files, 407,176 characters)
23:01:43  HTTP/1.1 504 Gateway Timeout    601 seconds
          Server: cloudflare, CF-RAY: ..., Retry-After: 120
23:01:44  openai._base_client  Retrying due to status code 504

Fifteen 504s across three runs, all between 600 and 602 seconds.

Why it reads as a hang

Three things stack up:

1. The OpenAI SDK retries twice on its own (max_retries=2, not overridden). Each retry re-uploads the 420KB prompt and spends another 600s. Those retries log at DEBUG.
2. _run_with_retries wraps that with 4 more attempts, so an exception can take about two hours to surface.
3. Nothing is logged between sending the request and getting a response.

The MCP client gives up long before those loops finish, so TOOL_COMPLETED never fires and the user gets nothing back. No error, no partial output, no log line pointing anywhere. I found the 504s by reading the whole log; they had been sitting there since the first failure.

Reproduction

1. Configure a custom OpenAI-compatible provider backed by a reasoning model.
2. Call codereview with relevant_files totalling ~100k tokens. Mine was 5-6 files, ~420,000 characters.
3. No TOOL_COMPLETED, blank result. logs/mcp_server.log shows the 504 at ~601s and the SDK retries after it.

Same endpoint and payload, tested outside PAL:

request	result
stream: false, ~113k tokens	HTTP 504 at ~601s
stream: true, ~113k tokens	first byte 12.5s, done 450s, finish_reason: stop

Streaming reached first byte in 4.6 to 12.5 seconds whatever the payload size. Every byte resets the proxy's idle timer.

The limit is time, not context

113k tokens against a 1M window. Nothing overflowed. The only question is how long a buffered body takes to build.

Suggested resolution

Streaming fixes it, but not as a global default. #134 ("O3 streaming=False for non-validated organizations") is the case that breaks: O3 through OpenRouter needs a verified organisation to stream at all.

An opt-in per provider or per model covers both:

• a supports_streaming capability in the model config (custom_models.json and friends), defaulting off, or
• an env flag like CUSTOM_STREAM=1 scoped to the custom provider

The streaming path is small: set stream: True, add stream_options: {"include_usage": True} so usage still arrives in the final chunk, accumulate delta.content, and take finish_reason from the last chunk that carries one. I run that locally as a patch and it has held up.

Two smaller findings, happy to split them out:

• providers/openai_compatible.py:643 returns content unchecked, and tools/workflow/workflow_mixin.py:1502 treats "" as falsy. A reasoning model can return HTTP 200 with content: "" and finish_reason: "length" when reasoning eats the output budget. That path produces a silent blank instead of an error or a retry.
• One log line either side of the API call turns this from a day of log archaeology into a glance.

Where the opt-in belongs is your call. I have the reproduction if you want more detail on any part of it.