Skip to content

Commit b6d901c

Browse files
rustyconoverclaude
andcommitted
perf(http): size waitress's buffers for Arrow bodies
Waitress's defaults are tuned for small web payloads and are actively hostile to Arrow: inbuf_overflow (512 KiB) spools a larger request body to a temp file before the app sees it, outbuf_overflow (1 MiB) does the same for responses, and recv_bytes (8 KiB) assembles a 6 MiB body from ~800 recv() calls. A single emitted batch is routinely megabytes, so every one of those thresholds sits on the hot path. Measured with a 6 MiB batch per tick: 0.81s per scan compressed, 119.9s uncompressed -- a 147x cliff, purely because the compressed body lands under outbuf_overflow and the plain one does not. Compression was masking an untuned transport, and a batch large enough to exceed the threshold *after* compression falls off the same cliff with no warning. These are the values vgi_rpc.http.server.serve_http already uses for this exact reason; the worker serves waitress directly and never inherited them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent a550612 commit b6d901c

1 file changed

Lines changed: 27 additions & 0 deletions

File tree

‎vgi/worker.py‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1623,11 +1623,38 @@ def _run_http(
16231623
# Symptom: waitress logs ``Task queue depth is N`` at INFO when the
16241624
# accept queue grows past 0. Pass ``--http-threads`` to size for
16251625
# the workload's expected concurrency.
1626+
# Waitress's buffer defaults are tuned for small web payloads and are
1627+
# actively hostile to Arrow bodies:
1628+
#
1629+
# inbuf_overflow 512 KiB -- a larger REQUEST body is spooled to a
1630+
# temp file on disk and read back before the app sees it.
1631+
# outbuf_overflow 1 MiB -- likewise for a larger RESPONSE body.
1632+
# recv_bytes 8 KiB -- a 6 MiB body assembled from ~800 recv()
1633+
# calls plus that many buffer appends.
1634+
#
1635+
# A single emitted batch is routinely megabytes, so every one of those
1636+
# thresholds is crossed on the hot path. Measured on this worker with a
1637+
# 6 MiB batch per tick: 0.81s per scan compressed, but 119.9s with
1638+
# compression disabled -- a 147x cliff, because the compressed body
1639+
# happens to land under `outbuf_overflow` and the uncompressed one does
1640+
# not. Compression was masking an untuned transport; a batch large
1641+
# enough to exceed the threshold *after* compression falls off the same
1642+
# cliff with no warning.
1643+
#
1644+
# These are the values `vgi_rpc.http.server.serve_http` already uses for
1645+
# exactly this reason; the worker serves waitress directly and so never
1646+
# inherited them.
1647+
_io_chunk = 1 << 20 # 1 MiB socket read/write chunks
1648+
_body_buffer = 64 << 20 # keep multi-MiB Arrow bodies in RAM
16261649
serve_kwargs: dict[str, Any] = {
16271650
"host": host,
16281651
"port": port,
16291652
"_quiet": True,
16301653
"asyncore_use_poll": True,
1654+
"inbuf_overflow": _body_buffer,
1655+
"outbuf_overflow": _body_buffer,
1656+
"recv_bytes": _io_chunk,
1657+
"send_bytes": _io_chunk,
16311658
}
16321659
if http_threads is not None:
16331660
serve_kwargs["threads"] = http_threads

0 commit comments

Comments
 (0)