Skip to content

perf(registry): ring doorbells from a cache instead of scanning every… - #59

Merged
benaliabderrahmane merged 1 commit into
develfrom
perf/cheap-doorbell-ring
Aug 19, 2026
Merged

benaliabderrahmane merged 1 commit into
develfrom
perf/cheap-doorbell-ring

Conversation

@benaliabderrahmane

Copy link
Copy Markdown
Owner

… slot

ring_doorbells runs after every registry mutation and walked [0, high_water) looking for doorbell slots. On a busy domain that is thousands of strided slot reads, each its own cache line, to find the handful of entries that are doorbells. Creating or destroying any node, publisher, subscription, service or client paid it.

Doorbells only come and go when a process starts or stops, which is far rarer than entity churn, so add doorbell_generation to the header and bump it only on an ENTRY_DOORBELL claim or release. Each ringing thread caches the resolved doorbell addresses and rebuilds only when that counter moves, so the common mutation rings straight from the vector with no scan at all.

doorbell_generation lands in the padding that already followed high_water_slot: sizeof(RegistryHeader) stays 64, slots still start at 64, and the total shm size is unchanged. A static_assert pins it there. REGISTRY_VERSION still goes to 5, because a v4 writer never bumps the counter and would leave a v5 reader holding a stale cache and missing wakeups, so the two must not share a domain.

Also shrink the doorbell socket's receive buffer. create_bound_socket sizes buffers for real payloads, but a doorbell datagram carries none: it only says "the registry changed", and rmw_wait drains the whole queue and re-reads the registry once. Bounding the queue lets the kernel coalesce a storm instead of piling up hundreds of datagrams for one re-read.

Removing the close-and-retry on EAGAIN was tried and reverted: LatchedTopicSurvivesAnUnresponsiveParticipant fails without it. AF_UNIX datagrams stay charged to the sender's buffer until the receiver consumes them, so a peer that never drains still starves sends to healthy peers. A smaller receive buffer reduces how much one peer can pin but does not remove the effect, so the reset stays, now shared by both send paths in ring_one().

Closes #53

… slot

ring_doorbells runs after every registry mutation and walked [0, high_water)
looking for doorbell slots. On a busy domain that is thousands of strided slot
reads, each its own cache line, to find the handful of entries that are
doorbells. Creating or destroying any node, publisher, subscription, service or
client paid it.

Doorbells only come and go when a process starts or stops, which is far rarer
than entity churn, so add doorbell_generation to the header and bump it only on
an ENTRY_DOORBELL claim or release. Each ringing thread caches the resolved
doorbell addresses and rebuilds only when that counter moves, so the common
mutation rings straight from the vector with no scan at all.

doorbell_generation lands in the padding that already followed high_water_slot:
sizeof(RegistryHeader) stays 64, slots still start at 64, and the total shm size
is unchanged. A static_assert pins it there. REGISTRY_VERSION still goes to 5,
because a v4 writer never bumps the counter and would leave a v5 reader holding
a stale cache and missing wakeups, so the two must not share a domain.

Also shrink the doorbell socket's receive buffer. create_bound_socket sizes
buffers for real payloads, but a doorbell datagram carries none: it only says
"the registry changed", and rmw_wait drains the whole queue and re-reads the
registry once. Bounding the queue lets the kernel coalesce a storm instead of
piling up hundreds of datagrams for one re-read.

Removing the close-and-retry on EAGAIN was tried and reverted:
LatchedTopicSurvivesAnUnresponsiveParticipant fails without it. AF_UNIX
datagrams stay charged to the sender's buffer until the receiver consumes them,
so a peer that never drains still starves sends to healthy peers. A smaller
receive buffer reduces how much one peer can pin but does not remove the
effect, so the reset stays, now shared by both send paths in ring_one().

Closes #53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves registry mutation performance by avoiding a full slot scan on every ring_doorbells() call, and reduces doorbell wakeup queueing pressure by shrinking the doorbell socket receive buffer.

Changes:

  • Add doorbell_generation to the shared-memory header and rebuild per-thread doorbell address caches only when doorbells are added/removed.
  • Refactor doorbell ringing to reuse cached sockaddr_un targets and centralize the EAGAIN reset/retry logic.
  • Reduce doorbell SO_RCVBUF and add a regression test covering cache invalidation when a doorbell registers after cache warm-up.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
rmw_unix_socket_cpp/test/test_registry.cpp Adds regression test for doorbell cache invalidation; adds socket headers.
rmw_unix_socket_cpp/src/rmw_init.cpp Shrinks doorbell socket receive buffer via setsockopt(SO_RCVBUF).
rmw_unix_socket_cpp/src/registry.hpp Bumps registry version to v5; adds doorbell_generation to header with layout guard.
rmw_unix_socket_cpp/src/registry.cpp Implements cached doorbell address ringing keyed by doorbell_generation; adds ring_one() helper and bumps doorbell_generation on doorbell add/remove/reclaim.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +443 to +447
struct RingState
{
int fd = -1;
~RingFd() {if (fd >= 0) {close(fd);}}
uint32_t built_at = 0;
bool primed = false;
Comment on lines +309 to +314
const char * path_a = "/tmp/rmw_uds_test_doorbell_a.sock";
const char * path_b = "/tmp/rmw_uds_test_doorbell_b.sock";
const int fd_a = bind_doorbell(path_a);
const int fd_b = bind_doorbell(path_b);
ASSERT_GE(fd_a, 0);
ASSERT_GE(fd_b, 0);
@benaliabderrahmane
benaliabderrahmane merged commit a0eab40 into devel Aug 19, 2026
4 checks passed
@benaliabderrahmane

Copy link
Copy Markdown
Owner Author

Backed out of devel. This PR was merged by mistake; devel was rebuilt without it (141f8bd → 450450b, dropping merge a0eab40) and force-pushed, so GitHub's "Merged" badge above no longer reflects devel's content — none of this branch's changes are on devel.

Context: the slot-scan cost this PR removes from ring_doorbells is addressed differently by #60, whose interest-scoped doorbell changes the same ring path; landing both would have meant reconciling two overlapping mechanisms. The branch perf/cheap-doorbell-ring (f2be7d3) is untouched if any of it is worth revisiting after #60 settles.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants