Fixed 300s worker timeout kills legitimate long-context requests and crashes the worker (BrokenPipeError)

[Hi5808]

Summary

rkllama's server has a hardcoded 300-second timeout on worker responses (No response received by the Worker of the model ... in 300 seconds). This timeout does not scale with prompt length. On long-context requests that legitimately take longer than 300s to prefill, the server aborts the request and the worker process crashes with a BrokenPipeError while trying to report its (successful) result back through the IPC pipe — because the parent has already closed its end after the timeout fired.

This isn't a hang or deadlock: the worker actually finishes the computation, it just can't report back in time.

Environment

• Board: Radxa ROCK 5B+ (RK3588), Armbian 26.8.1, kernel 6.1.115-vendor-rk35xx
• rkllama version: 0.0.75
• rkllm-runtime: 1.3.0
• Model: dulimov/Qwen3-1.7B-rk3588-1.2.1-unsloth-16k (w8a8_g128-opt-0-hybrid-ratio-0.0, converted with rkllm-toolkit 1.2.1b1, max_context_limit: 16384)

Steps to reproduce

1. Load a model converted with a large max_context_limit (e.g. 16384).
2. Send a /api/chat request with a prompt long enough that prefill alone takes >300s. On this hardware, prefill scaling for the 1.7B model above was:
   • 1024 tokens → ~15s
   • 4096 tokens → ~71s
   • 8192 tokens → ~226s
   • 16000 tokens → exceeds 300s (extrapolated ~10-12 minutes based on the above trend)
3. Observe the server logs.

Actual behavior

2026-08-11 13:08:35,700 - rkllama.worker - INFO - Running inference for model Qwen3-1.7B-rk3588-16k...
2026-08-11 13:13:35,765 - rkllama.server_utils - ERROR - No response received by the Worker of the model Qwen3-1.7B-rk3588-16k in 300 seconds.
2026-08-11 13:13:35,766 - rkllama.worker - INFO - Aborting inference for model Qwen3-1.7B-rk3588-16k...
2026-08-11 13:13:35,895 - rkllama.worker - ERROR - Failed executing task the worker for model 'Qwen3-1.7B-rk3588-16k': [Errno 32] Broken pipe
Process Process-9:
Traceback (most recent call last):
  File ".../rkllama/api/worker.py", line 321, in run_rkllm_worker
    child_conn.send((WORKER_TASK_FINISHED, prompt_token_count, token_count, prompt_eval, eval))
  ...
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  ...
  File ".../rkllama/api/worker.py", line 367, in run_rkllm_worker
    child_conn.send(WORKER_TASK_ERROR)
  ...
BrokenPipeError: [Errno 32] Broken pipe

Note the worker had reached the WORKER_TASK_FINISHED send call — i.e. inference had actually completed — when the pipe write failed, because the parent side had already closed it after the 300s timeout.

The HTTP client does receive a 200 OK response (from the abort path), but with no useful content, and the worker process for that model dies and needs to be respawned on the next request.

Expected behavior

Either:

• The timeout should scale with (or otherwise account for) prompt/context length, since prefill time is highly length-dependent, or
• The timeout should be configurable (server flag or Modelfile parameter), or
• At minimum, the abort path shouldn't leave the worker in a state where a subsequent legitimate completion attempts to write to an already-closed pipe and crashes — e.g. check whether the pipe is still open / catch BrokenPipeError specifically around the result-send calls in worker.py so a late-but-successful completion doesn't produce a confusing crash trace.

Additional context

This same failure mode is very plausibly what caused an earlier, apparently-unrelated observation on our end: a request hanging for 20+ minutes under heavy unrelated CPU load elsewhere on the machine, which we initially attributed to CPU contention starving the NPU worker. Reproducing this cleanly (with a long-context prompt and zero external CPU contention) at 16k tokens suggests the contention wasn't the root cause of the failure itself — it just pushed an otherwise-borderline-length request further past the same fixed 300s wall. So this fixed timeout may be responsible for more of the "flaky under load" reports than it appears at first.

Full writeup with the underlying benchmark data: https://claude.ai/code/artifact/09b27c30-c837-4204-8941-1d107ee75c9d