Surfaced while closing #899. With the http benchmark server serving concurrently (#902), the same binary measures ~14-15.6k req/s under the green backend and ~16.8-16.9k under PITH_GREEN=0 — the os-thread backend wins by ~10-15% on this workload. Filed with the measurements so the next attempt starts where this one stopped.
what is established (all same-session, interleaved where it matters)
- The gap is per-request cpu in the green path: 103µs/request against 93µs for threads under identical load. Flat across
PITH_GREEN_WORKERS=2/4/8, so it is not worker-count queuing.
- The data path is already lean: exactly 1
sendto + 1 recvfrom per request on both backends; green adds only ~34 epoll_wait calls/second (heavily batched) and ~50 epoll_ctl/s. The overhead is user-space bookkeeping, not syscalls.
- Sequential latency is equal (135µs green vs 133µs threads on a single connection), so the cost only appears under concurrency: it is the park/arm/reactor/resume round trip a task pays when the next keepalive request has not arrived by the time it reads.
- Why threads win: a blocked
recvfrom is woken by the data itself — the wake is the delivery, zero runtime bookkeeping. They pay heavy io-registry contention (0.35 sched_yield + 0.5 futex per request) and still come out ahead.
- Oddity worth understanding: removing
tcp_set_timeout makes green worse (11k, 125µs/request) — the deadline path is currently the faster read path.
what was tried and refuted
Bounded spin-before-park at the fd level (the #780 lesson one level down: retry the nonblocking read 5µs before yielding to the reactor). Interleaved A/B, four rounds: the spin lost every round (-20% throughput, +30% cpu/request). On two cores the peer producing the next request is the load generator itself, so spinning displaces exactly the work that would deliver the data. #780's channel spin won because the partner ran independently. Do not retry this shape on a small box; it might still hold on one with spare cores, but it cannot be validated here.
the direction the thread path actually points
Not spinning — making the wake the delivery. Today an idle green worker parks on its own condvar while a dedicated reactor does epoll_wait and hands tasks over (park, arm, wake, enqueue, resume — five steps where the kernel gives threads one). The established design for closing exactly this gap is netpoller integration: an idle worker becomes the poller, calling epoll_wait itself, so socket readiness wakes the precise thread that will run the task, one hop. That is a structural change to netpoll/green ownership, sized well beyond a prototype, which is why this is an issue and not a PR.
measurement discipline for whoever picks this up
The box's cpu/request degrades ~2x across a sustained multi-minute wrk batch (109 -> 188µs for the same binary — e2 burst decay), so absolute numbers from consecutive runs are launch-cadence artifacts. Only interleaved same-round comparisons are valid, and the sequential single-connection probe (one keepalive connection, p50 over thousands of requests) is deterministic to ±2µs where wrk is noise.
Surfaced while closing #899. With the http benchmark server serving concurrently (#902), the same binary measures ~14-15.6k req/s under the green backend and ~16.8-16.9k under
PITH_GREEN=0— the os-thread backend wins by ~10-15% on this workload. Filed with the measurements so the next attempt starts where this one stopped.what is established (all same-session, interleaved where it matters)
PITH_GREEN_WORKERS=2/4/8, so it is not worker-count queuing.sendto+ 1recvfromper request on both backends; green adds only ~34epoll_waitcalls/second (heavily batched) and ~50epoll_ctl/s. The overhead is user-space bookkeeping, not syscalls.recvfromis woken by the data itself — the wake is the delivery, zero runtime bookkeeping. They pay heavy io-registry contention (0.35sched_yield+ 0.5futexper request) and still come out ahead.tcp_set_timeoutmakes green worse (11k, 125µs/request) — the deadline path is currently the faster read path.what was tried and refuted
Bounded spin-before-park at the fd level (the #780 lesson one level down: retry the nonblocking read
5µs before yielding to the reactor). Interleaved A/B, four rounds: the spin lost every round (-20% throughput, +30% cpu/request). On two cores the peer producing the next request is the load generator itself, so spinning displaces exactly the work that would deliver the data. #780's channel spin won because the partner ran independently. Do not retry this shape on a small box; it might still hold on one with spare cores, but it cannot be validated here.the direction the thread path actually points
Not spinning — making the wake the delivery. Today an idle green worker parks on its own condvar while a dedicated reactor does
epoll_waitand hands tasks over (park, arm, wake, enqueue, resume — five steps where the kernel gives threads one). The established design for closing exactly this gap is netpoller integration: an idle worker becomes the poller, callingepoll_waititself, so socket readiness wakes the precise thread that will run the task, one hop. That is a structural change to netpoll/green ownership, sized well beyond a prototype, which is why this is an issue and not a PR.measurement discipline for whoever picks this up
The box's cpu/request degrades ~2x across a sustained multi-minute wrk batch (109 -> 188µs for the same binary — e2 burst decay), so absolute numbers from consecutive runs are launch-cadence artifacts. Only interleaved same-round comparisons are valid, and the sequential single-connection probe (one keepalive connection, p50 over thousands of requests) is deterministic to ±2µs where wrk is noise.