Repository navigation
perf(registry): ring doorbells from a cache instead of scanning every… - #59
Conversation
… slot ring_doorbells runs after every registry mutation and walked [0, high_water) looking for doorbell slots. On a busy domain that is thousands of strided slot reads, each its own cache line, to find the handful of entries that are doorbells. Creating or destroying any node, publisher, subscription, service or client paid it. Doorbells only come and go when a process starts or stops, which is far rarer than entity churn, so add doorbell_generation to the header and bump it only on an ENTRY_DOORBELL claim or release. Each ringing thread caches the resolved doorbell addresses and rebuilds only when that counter moves, so the common mutation rings straight from the vector with no scan at all. doorbell_generation lands in the padding that already followed high_water_slot: sizeof(RegistryHeader) stays 64, slots still start at 64, and the total shm size is unchanged. A static_assert pins it there. REGISTRY_VERSION still goes to 5, because a v4 writer never bumps the counter and would leave a v5 reader holding a stale cache and missing wakeups, so the two must not share a domain. Also shrink the doorbell socket's receive buffer. create_bound_socket sizes buffers for real payloads, but a doorbell datagram carries none: it only says "the registry changed", and rmw_wait drains the whole queue and re-reads the registry once. Bounding the queue lets the kernel coalesce a storm instead of piling up hundreds of datagrams for one re-read. Removing the close-and-retry on EAGAIN was tried and reverted: LatchedTopicSurvivesAnUnresponsiveParticipant fails without it. AF_UNIX datagrams stay charged to the sender's buffer until the receiver consumes them, so a peer that never drains still starves sends to healthy peers. A smaller receive buffer reduces how much one peer can pin but does not remove the effect, so the reset stays, now shared by both send paths in ring_one(). Closes #53
There was a problem hiding this comment.
Pull request overview
Improves registry mutation performance by avoiding a full slot scan on every ring_doorbells() call, and reduces doorbell wakeup queueing pressure by shrinking the doorbell socket receive buffer.
Changes:
- Add
doorbell_generationto the shared-memory header and rebuild per-thread doorbell address caches only when doorbells are added/removed. - Refactor doorbell ringing to reuse cached
sockaddr_untargets and centralize the EAGAIN reset/retry logic. - Reduce doorbell
SO_RCVBUFand add a regression test covering cache invalidation when a doorbell registers after cache warm-up.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| rmw_unix_socket_cpp/test/test_registry.cpp | Adds regression test for doorbell cache invalidation; adds socket headers. |
| rmw_unix_socket_cpp/src/rmw_init.cpp | Shrinks doorbell socket receive buffer via setsockopt(SO_RCVBUF). |
| rmw_unix_socket_cpp/src/registry.hpp | Bumps registry version to v5; adds doorbell_generation to header with layout guard. |
| rmw_unix_socket_cpp/src/registry.cpp | Implements cached doorbell address ringing keyed by doorbell_generation; adds ring_one() helper and bumps doorbell_generation on doorbell add/remove/reclaim. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| struct RingState | ||
| { | ||
| int fd = -1; | ||
| ~RingFd() {if (fd >= 0) {close(fd);}} | ||
| uint32_t built_at = 0; | ||
| bool primed = false; |
| const char * path_a = "/tmp/rmw_uds_test_doorbell_a.sock"; | ||
| const char * path_b = "/tmp/rmw_uds_test_doorbell_b.sock"; | ||
| const int fd_a = bind_doorbell(path_a); | ||
| const int fd_b = bind_doorbell(path_b); | ||
| ASSERT_GE(fd_a, 0); | ||
| ASSERT_GE(fd_b, 0); |
|
Backed out of Context: the slot-scan cost this PR removes from |
… slot
ring_doorbells runs after every registry mutation and walked [0, high_water) looking for doorbell slots. On a busy domain that is thousands of strided slot reads, each its own cache line, to find the handful of entries that are doorbells. Creating or destroying any node, publisher, subscription, service or client paid it.
Doorbells only come and go when a process starts or stops, which is far rarer than entity churn, so add doorbell_generation to the header and bump it only on an ENTRY_DOORBELL claim or release. Each ringing thread caches the resolved doorbell addresses and rebuilds only when that counter moves, so the common mutation rings straight from the vector with no scan at all.
doorbell_generation lands in the padding that already followed high_water_slot: sizeof(RegistryHeader) stays 64, slots still start at 64, and the total shm size is unchanged. A static_assert pins it there. REGISTRY_VERSION still goes to 5, because a v4 writer never bumps the counter and would leave a v5 reader holding a stale cache and missing wakeups, so the two must not share a domain.
Also shrink the doorbell socket's receive buffer. create_bound_socket sizes buffers for real payloads, but a doorbell datagram carries none: it only says "the registry changed", and rmw_wait drains the whole queue and re-reads the registry once. Bounding the queue lets the kernel coalesce a storm instead of piling up hundreds of datagrams for one re-read.
Removing the close-and-retry on EAGAIN was tried and reverted: LatchedTopicSurvivesAnUnresponsiveParticipant fails without it. AF_UNIX datagrams stay charged to the sender's buffer until the receiver consumes them, so a peer that never drains still starves sends to healthy peers. A smaller receive buffer reduces how much one peer can pin but does not remove the effect, so the reset stays, now shared by both send paths in ring_one().
Closes #53