Skip to content

fix: replay TRANSIENT_LOCAL cache to late-joining subscribers - #2

Merged
benaliabderrahmane merged 1 commit into
mainfrom
feature/transient-local-wait-replay
Jun 5, 2026
Merged

benaliabderrahmane merged 1 commit into
mainfrom
feature/transient-local-wait-replay

Conversation

@benaliabderrahmane

Copy link
Copy Markdown
Owner

Closes #1

Summary

The publisher previously only replayed its cached messages inside rmw_publish, so a subscriber that joined after the publisher had gone idle never received the cache. In production this manifested as late-joining subscribers (ros2 topic echo, bridge consumers, etc.) silently missing latched topics such as camera_list.

This change tracks TRANSIENT_LOCAL publishers on the context and drives cache replay from rmw_wait whenever the registry generation changes — the existing publish-side replay still works; the wait-side path covers the idle-publisher case. Both paths cooperate via known_subscriber_paths (whichever marks a path first, the other no-ops).

Implementation notes

  • UdsContext gains transient_local_pubs (+ a mutex). Publishers register on create, deregister on destroy.
  • The wait-side replay holds transient_local_pubs_mutex across the whole loop so rmw_destroy_publisher (which takes the same mutex then deletes) cannot free a publisher mid-iteration.
  • No change to the publish-side replay logic; this strictly adds the second trigger.

Test plan

  • TransientLocalReplayOnWaitNoSubsequentPublish — publisher publishes N messages then goes idle; subscriber joins, calls rmw_wait once, must receive all N cached messages without a fresh publish.
  • TransientLocalReplayOnWaitDepthOne — same flow with depth=1.
  • Both tests fail on main and pass on this branch (verified via git stash + rebuild + ctest).
  • Full ctest suite: 12/12 pass.

@benaliabderrahmane benaliabderrahmane added the bug Something isn't working label Jun 1, 2026
@benaliabderrahmane

Copy link
Copy Markdown
Owner Author

Reviewed PR #2 (head feature/transient-local-wait-replay, single commit 0bd2c1f) against base main. Verified every cited file:line via git show feature/transient-local-wait-replay:<file> since the working tree has diverged.

Overall: a focused, correct, well-scoped improvement. The new wait-side TRANSIENT_LOCAL replay genuinely fixes the same-process late-joiner case (component containers / multi-node processes), the lifetime invariant is sound (transient_local_pubs_mutex held across the loop keeps pub pointers alive against rmw_destroy_publisher), there is no lock-order inversion (transient_local_pubs_mutex -> sub_cache_mutex -> cache_mutex is consistent with the publish path), the failing-test-first reproducer is genuine (on main rmw_wait.cpp has no transient_local_pubs reference, so the test fails; on the branch it passes), and CI is green on jazzy/kilted/rolling. Approving on its own merits.

One thing to fix before merge (non-blocking but please address the body, or file a follow-up):

  1. PR body overstates coverage. The headline production scenario — an idle pure-publisher process whose latched topic is missed by a late ros2 topic echo/bridge across processes — is NOT delivered by this design. ctx is resolved only from subscriptions[0]/services[0]/clients[0]->context (rmw_wait.cpp:162-165), i.e. the wait-caller's own process. A pure publisher process that subscribes to nothing never calls rmw_wait with a sub/srv/client, so ctx stays nullptr and the replay block (rmw_wait.cpp:168+) is skipped even if it waits on a guard condition; the late subscriber in a different process iterates its own (empty) transient_local_pubs. Both added tests share one context (test_base.hpp single node), so only the same-process path is exercised. Please correct the body to describe the same-process scope, or add a cross-process integration test.

Worth noting (pre-existing patterns this PR extends, fine to defer to the scheduled perf/churn PRs):

  • Durability gap: mark-before-send + ignored send_to return (rmw_wait.cpp:201 inserts before :203 send_to). A transiently-failed replay (ENOBUFS/EAGAIN) permanently marks the path and is never retried; since the idle-publisher path has no subsequent publish to mask the drop, the late sub loses the latched samples. Mirrors rmw_publisher.cpp:244-245 but is newly the sole delivery mechanism here.
  • Latency: replayed bytes can be deferred one wait cycle when another entity is already ready — the step-4 drain is guarded by if (!something_ready) (rmw_wait.cpp:277) and the step-1 drain runs before ctx resolution. Not data loss.
  • No durability filtering on replay (rmw_wait.cpp:182-194 replays to VOLATILE subs too); registry already carries durability, so filtering is feasible. Pre-existing on the publish side.
  • Perf/leak: per-generation O(max_entries) registry scan per idle TL pub (rmw_wait.cpp:182), and known_subscriber_paths now grows from a second (wait) writer with no pruning under churn.

Simplicity nits (CLAUDE.md sections 2/3, optional):

  • The sub-path refresh block (rmw_wait.cpp:180-194) is a near-verbatim copy of rmw_publisher.cpp:215-235 — consider a shared UdsPublisher::refresh_subscriber_paths() helper.
  • Divergent replay slices: publish-side skips the last cache entry (i + 1 < size, rmw_publisher.cpp:248, sent separately after) while wait-side sends the full cache (rmw_wait.cpp:202). Both correct, but the structural divergence is a drift hazard.

Test coverage to consider adding: two-path dedup cooperation (publish + wait replay to the same sub across one generation change), >1 TL publisher, and the cross-process path.

LGTM to merge as an incremental improvement once the body is corrected.

The publisher previously only replayed its cached messages inside
rmw_publish, so a subscriber that joined after the publisher had gone
idle never received the cache. In production this manifested as
late-joining subscribers (e.g. `ros2 topic echo`, bridge consumers)
silently missing latched topics such as camera_list.

Track TRANSIENT_LOCAL publishers on the context and drive cache replay
from rmw_wait whenever the registry generation changes. The mutex is
held across the loop so rmw_destroy_publisher cannot free a publisher
mid-iteration. known_subscriber_paths still deduplicates: whichever
path (publish or wait) runs first marks the path, the other no-ops.

Tests:
- TransientLocalReplayOnWaitNoSubsequentPublish: publisher publishes
  N messages then goes idle; subscriber joins, calls rmw_wait once,
  must receive all N cached messages without a fresh publish.
- TransientLocalReplayOnWaitDepthOne: same flow with depth=1.

Both tests fail on HEAD and pass with this change.
@benaliabderrahmane
benaliabderrahmane force-pushed the feature/transient-local-wait-replay branch from 0bd2c1f to a5b7503 Compare June 5, 2026 18:01
@benaliabderrahmane
benaliabderrahmane merged commit bc5cc63 into main Jun 5, 2026
3 checks passed
@benaliabderrahmane
benaliabderrahmane deleted the feature/transient-local-wait-replay branch June 17, 2026 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TRANSIENT_LOCAL: late-joining subscribers do not receive cached messages

1 participant