You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6d9793c
Browse filesBrowse the repository at this point in the historyBrowse files
fix(registry): don't free a slot before its teardown finishes (#81)
registry_remove and registry_cleanup_stale CASed the slot state to EMPTY and
only then ran teardown_slot. In that gap another process's try_add_once could
claim the slot and write its entry, which the late teardown then zeroed (and
whose socket file it unlinked). The live endpoint vanished from the graph and
a nameless ghost stayed behind: state=SERVICE, pid 0, empty name and type,
which cleanup_stale never reclaims. rmw_get_service_names_and_types then
returned ('', ['']) and foxglove_bridge aborted ("Topic or service name can't
be empty"). Seen in a 10-robot fleet sharing one PID namespace, where many
processes sweep the same dead spawner PIDs while others register at launch.
Hold the slot in ENTRY_RESERVED during teardown and store ENTRY_EMPTY after
it. Adders only claim EMPTY and readers already skip RESERVED, so the shm
layout is unchanged. registry_remove also returns early on RESERVED.
Test: StaleReapNeverClobbersConcurrentAdd fails on main (~1,400 of 20,000
live entries wiped per run, 3/3 runs) and passes with the fix.
Copy file name to clipboardExpand all lines: rmw_unix_socket_cpp/DESIGN.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -394,7 +394,7 @@ The release store pairs with the acquire load every reader does on `state`. Any
394
394
395
395
**Bump the generation counter, then ring the doorbells.** Finally the writer does `header->generation.fetch_add(1)`. The generation counter is the table's change signal. Publishers, services, and clients cache their lookup results and re-scan only when the generation moves, so bumping it here tells every cached reader that the graph changed and its cache is stale. Immediately after the bump, the writer sends one octet to every registered doorbell (`ring_doorbells`), which wakes any process blocked in `rmw_wait` so it re-reads the counter. The order is load-bearing: ring strictly after the bump, paired with the wait side draining its doorbell strictly before reading the counter, is what makes a lost wakeup impossible (see The doorbell under Wait set). This is what turns a new registration into a graph event without any daemon: the table is the state, the doorbell is the edge.
396
396
397
-
**The slot index is remembered, so removal needs no scan.**`registry_add` returns the slot index, and the owning entity stores it. Removal (`registry_remove`) indexes straight to that slot and CASes its `state`back to`ENTRY_EMPTY`. There is no second scan to find the entry on the way out, which keeps teardown cheap and makes removal symmetric with the single-CAS claim used on the way in.
397
+
**The slot index is remembered, so removal needs no scan.**`registry_add` returns the slot index, and the owning entity stores it. Removal (`registry_remove`) indexes straight to that slot, CASes its `state`to `ENTRY_RESERVED`, tears it down, and only then stores`ENTRY_EMPTY`. There is no second scan to find the entry on the way out, which keeps teardown cheap and makes removal symmetric with the single-CAS claim used on the way in.
398
398
399
399
**Failure path.** If the scan finds no empty slot, `registry_add` runs the stale-cleanup pass once (reclaiming slots whose owning PID is dead) and retries the scan a single time. If the table is still full after that, the add fails and returns -1. There is no resize and no spinning.
400
400
@@ -436,9 +436,9 @@ Services and clients cache the same way against the same generation counter; the
436
436
437
437
Two failure modes leave entries behind: a clean shutdown, and a process that dies without running its destructors. They are handled separately because only one of them can be trusted to tell the truth about itself.
438
438
439
-
**Clean shutdown.** When an endpoint is destroyed normally, it removes its own slot. Because each entity records the slot index it was assigned at registration, removal needs no scan: it indexes straight to its slot. `registry_remove` reads the slot's current `state`, then attempts a single compare-and-swap from that state value to `ENTRY_EMPTY`. The CAS is what makes the removal safe under concurrency: exactly one caller can win it, and the winner owns the teardown. If the CAS fails, someone else already emptied the slot, so the function returns without touching anything.
439
+
**Clean shutdown.** When an endpoint is destroyed normally, it removes its own slot. Because each entity records the slot index it was assigned at registration, removal needs no scan: it indexes straight to its slot. `registry_remove` reads the slot's current `state`, then attempts a single compare-and-swap from that state value to `ENTRY_RESERVED`. The CAS is what makes the removal safe under concurrency: exactly one caller can win it, and the winner owns the teardown. If the CAS fails (or the slot is already `ENTRY_EMPTY` or `ENTRY_RESERVED`), someone else is emptying it, so the function returns without touching anything.
440
440
441
-
The teardown itself runs under the slot's seqlock (the same odd/even protocol used on the write side). `teardown_slot` bumps `seq` to odd, copies out the socket path, zeroes every payload field, then bumps `seq` back to even. Any reader that was mid-snapshot sees the odd `seq` and retries, so it can never read a half-erased slot. The socket file is unlinked after the seqlock closes, because a filesystem operation does not need to be observable atomically by other processes. Finally `generation` is incremented so every cached reader learns the graph changed. The order matters: `state` flips to `ENTRY_EMPTY` first (no new reader can match the slot), the payload is wiped under the seqlock, and only then does `generation` move.
441
+
The teardown itself runs under the slot's seqlock (the same odd/even protocol used on the write side). `teardown_slot` bumps `seq` to odd, copies out the socket path, zeroes every payload field, then bumps `seq` back to even. Any reader that was mid-snapshot sees the odd `seq` and retries, so it can never read a half-erased slot. The socket file is unlinked after the seqlock closes, because a filesystem operation does not need to be observable atomically by other processes. Finally `generation` is incremented so every cached reader learns the graph changed. The order matters: `state` flips to `ENTRY_RESERVED` first (no new reader can match the slot, and no adder can claim it), the payload is wiped under the seqlock, `state` is set to `ENTRY_EMPTY`, and only then does `generation` move. Freeing the slot before the wipe would let a concurrent add claim it in between and have its fresh entry and socket file erased by the late teardown.
442
442
443
443
**Crash.** A process killed by SIGKILL, OOM, or a container teardown never reaches `registry_remove`. Its slot keeps a stale socket path that no longer accepts connections. There is no destructor to trust, so the registry infers death from the operating system instead. `registry_cleanup_stale` walks the table, and for each live slot it takes a seqlock snapshot and checks the owning PID by stat-ing `/proc/<pid>`. The decision is deliberately conservative:
444
444
@@ -448,7 +448,7 @@ The teardown itself runs under the slot's seqlock (the same odd/even protocol us
448
448
449
449
The bias toward keeping is intentional. The cost of wrongly keeping a dead slot is one wasted slot until the next pass; the cost of wrongly reclaiming a live slot is tearing down a working endpoint. A `pid` of 0 is also never reclaimed, since that value only appears in the transient snapshot of a slot whose payload has not been written yet, never in a committed entry.
450
450
451
-
The dangerous window is between vetting a slot and tearing it down. The owner could exit, the slot could be reclaimed by a clean remove, and a different process could re-add an entry of the same type into the same slot, all before this pass acts. This is the classic ABA problem: the slot looks identical but is a different entry, and the `state` value alone cannot tell them apart. The seqlock counter closes the gap. The snapshot captures the slot's `seq` (`snap_seq`), and immediately before the reclaim CAS the pass re-reads `seq` and compares. Any intervening write (a remove or a re-add both go through the seqlock) advances the counter, so a mismatch means the slot is no longer the entry that was checked, and the pass skips it. Only when `seq` is unchanged does it CAS the observed `state` to `ENTRY_EMPTY`, log a one-line breadcrumb naming the slot, PID, node, and topic so an ungraceful exit is visible in the ROS log, run the same `teardown_slot`, and bump `generation`.
451
+
The dangerous window is between vetting a slot and tearing it down. The owner could exit, the slot could be reclaimed by a clean remove, and a different process could re-add an entry of the same type into the same slot, all before this pass acts. This is the classic ABA problem: the slot looks identical but is a different entry, and the `state` value alone cannot tell them apart. The seqlock counter closes the gap. The snapshot captures the slot's `seq` (`snap_seq`), and immediately before the reclaim CAS the pass re-reads `seq` and compares. Any intervening write (a remove or a re-add both go through the seqlock) advances the counter, so a mismatch means the slot is no longer the entry that was checked, and the pass skips it. Only when `seq` is unchanged does it CAS the observed `state` to `ENTRY_RESERVED`, log a one-line breadcrumb naming the slot, PID, node, and topic so an ungraceful exit is visible in the ROS log, run the same `teardown_slot`, store `ENTRY_EMPTY`, and bump `generation`.
452
452
453
453
**When it runs.** There are two triggers.
454
454
@@ -476,7 +476,7 @@ So the hot paths avoid locks entirely. They rely on the fact that `std::atomic<u
476
476
477
477
Three mechanisms replace the lock:
478
478
479
-
-**Claiming a slot is a single CAS.** To add an entry, a writer scans for an empty slot and flips its `state` from `ENTRY_EMPTY` to `ENTRY_RESERVED` with one atomic compare-and-swap. Exactly one writer wins. `ENTRY_RESERVED` is a private "claimed but not yet filled in" marker (see Registering an entry for the full read/write protocol). Removal is the mirror image: a CAS of `state`back to `ENTRY_EMPTY` decides who owns the teardown.
479
+
-**Claiming a slot is a single CAS.** To add an entry, a writer scans for an empty slot and flips its `state` from `ENTRY_EMPTY` to `ENTRY_RESERVED` with one atomic compare-and-swap. Exactly one writer wins. `ENTRY_RESERVED` is a private "claimed but not yet filled in" marker (see Registering an entry for the full read/write protocol). Removal is the mirror image: a CAS of `state` to `ENTRY_RESERVED` decides who owns the teardown, and `ENTRY_EMPTY` is stored only once the slot is wiped.
480
480
481
481
-**Publishing and reading a slot is a seqlock.** A writer brackets the payload copy with an odd/even `seq` counter; a reader snapshots `seq` before and after the copy and retries on mismatch. The full protocol is described in Registering an entry. The point for locking is that readers never block a writer and writers never block a reader, and the bounded retry (`MAX_READ_RETRIES`) guarantees a persistent writer cannot spin a reader forever.
0 commit comments