Conversation
packet__write() unconditionally registered EPOLLOUT via mux__add_out() before attempting to write, then removed it again once the out queue drained. In the common case where the kernel accepts the write immediately this cost two epoll_ctl() syscalls per write cycle that the event loop never needed. Only register EPOLLOUT when a write actually blocks (EAGAIN with data still pending); the bridge connect-pending case still arms it explicitly. On a single-subscriber QoS0 routed workload this roughly doubled throughput (~295k -> ~548k msg/s here) and removed epoll_ctl (~9.6% of broker CPU) from the profile, with no change to the broker test suite results. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dean Gostiša <dean.gostisa@black.si>
Adds a small, self-contained throughput benchmark under misc/bench: a single-connection QoS-0 publisher and a multi publisher/subscriber driver, with a README documenting how to A/B two builds without being fooled by host drift (interleave trials; verify the two binaries actually differ; prefer the deterministic `strace -c -e epoll_ctl` count). Used to substantiate the EPOLLOUT-toggle change: epoll_ctl on the write path drops from ~1.67 per message to ~0, and single-core throughput rises by roughly 60-90% across a range of publisher/subscriber counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dean Gostiša <dean.gostisa@black.si>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Broker: avoid EPOLLOUT add/remove churn on every write
Summary
packet__write()(broker build) registers EPOLLOUT viamux__add_out()at thetop of the function and removes it again via
mux__remove_out()once the outqueue drains. In the common case — the socket accepts the write synchronously —
this costs two
epoll_ctl()syscalls per write cycle that the event loopnever needed.
This changes it to the standard level-triggered pattern: only register EPOLLOUT
when a write actually blocks (
EAGAINwith data still pending). The outgoingbridge
connect_pendingcase still arms EPOLLOUT explicitly, andpacket__queuealready attempts an eager write for every queued packet, so nothing can be left
waiting for an EPOLLOUT that was never armed.
Deterministic effect (
strace -c -e epoll_ctl, fixed 200k-message run)perfbefore:epoll_ctl= ~9.6% of broker CPU; after: absent from the profile.Throughput (interleaved A/B, broker pinned to one core, plain TCP, 64B QoS0)
(Trials alternated between two fixed binaries built from the parent commit and
this commit, to control for host drift.)
Correctness
delivered intact and in order over plain TCP, TLS (SSL_write WANT_WRITE),
and builtin WebSockets.
corruption.
pollbackend (WITH_EPOLL=no) exercised as well as epoll.bridge tests that cover the
connect_pendingpath).Second commit (optional)
The second commit adds a small throughput benchmark under
misc/bench/used toproduce the numbers above. Happy to drop it if you'd rather keep it out of tree.