You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Every registry mutation scans all slots and wakes every process #53
ring_doorbells() runs after every registry mutation (registry.cpp#L395-L470). It does two things that scale with the size of the system rather than with the size of the change:
Walks all slots from 0 to high_water_slot looking for doorbell entries. On a system with ~7400 slots in use and 159 processes that is 7400 slot reads to find 159 doorbells. The reads are strided 1168 bytes apart, so almost every one is a cache miss.
Sends a datagram to each doorbell, so all 159 processes wake up and re-check the registry.
Creating or destroying any node, publisher, subscription, service or client triggers this.
During startup, when many nodes register at once, we measured 127.8 mutations/s. With 159 processes that is roughly 20k wakeups/s across the machine, plus 127.8 x 7400 slot reads/s in the mutating processes. An 8-core machine drops to 0% idle and stays there until the launch settles.
There is a second effect in the EAGAIN path (registry.cpp#L452-L465). When a peer is slow to drain its doorbell, sendto fails, and the sender closes its socket, creates a new one and retries. That is three extra syscalls per slow peer per mutation, so a single saturated process makes every other process slower.
For context, this replaced a 200 ms poll in 1ad02b5. The doorbell is the better design and that commit fixed two real bugs. But the poll also happened to cap this work at 5 Hz, and removing the poll removed the cap.
Suggested fix
Keep doorbells in a compact structure, a small array or a bitmap in the header, so ringing is O(doorbells) instead of O(high_water_slot).
Give each doorbell socket a small SO_RCVBUF. The doorbell carries no payload, it is only an edge signal, so redundant datagrams are safe to drop. A small buffer lets the kernel coalesce them and the drain loop then reads one or two messages instead of hundreds.
Once the receive queue is bounded, EAGAIN unambiguously means a wakeup is already queued for that peer, so the right action is to skip it rather than recreate the socket.
If that is not enough, debouncing the generation check on the receiving side would bound what is left. The timeout contract fixed in 1ad02b5 constrains what rmw_wait returns, not how often it re-reads the registry, so a debounce there does not bring that bug back.
Fix in flight: PR #60 (feature/tl-latched-pull) removes both halves of the storm for plain pub/sub processes — TRANSIENT_LOCAL replay becomes pull-based (the late joiner reads the publisher's latched shm cache itself at rmw_create_subscription; the idle publisher is never woken), and the doorbell slot registers lazily, only from waits that hold a graph guard condition. At a 200-node launch, wake datagrams drop from ~100k to ~the number of graph-event consumers. Residual after merge: ring_doorbells still walks the slot prefix per mutation to find the (now few) registered doorbells — ~µs, but if this issue is kept open post-merge it should be retitled to that residual.
Activity
benaliabderrahmane commentedon Aug 13, 2026
Fix in flight: PR #60 (
feature/tl-latched-pull) removes both halves of the storm for plain pub/sub processes — TRANSIENT_LOCAL replay becomes pull-based (the late joiner reads the publisher's latched shm cache itself atrmw_create_subscription; the idle publisher is never woken), and the doorbell slot registers lazily, only from waits that hold a graph guard condition. At a 200-node launch, wake datagrams drop from ~100k to ~the number of graph-event consumers. Residual after merge:ring_doorbellsstill walks the slot prefix per mutation to find the (now few) registered doorbells — ~µs, but if this issue is kept open post-merge it should be retitled to that residual.