Skip to content

registry_cleanup_stale runs on every graph query, costing one stat() per live entry #52

Closed

Description

@benaliabderrahmane

query_all() calls registry_cleanup_stale() before every graph query (rmw_graph.cpp#L46-L58). It has 13 call sites, so this covers rmw_get_node_names, rmw_count_publishers, rmw_count_subscribers, rmw_get_topic_names_and_types, rmw_get_service_names_and_types and the rest.

registry_cleanup_stale() walks every non-empty slot, snapshots 1164 bytes from it, and calls stat("/proc/<pid>") to check whether the owner is still alive (registry.cpp#L359-L371).

Reclaiming slots owned by dead processes is garbage collection. Running it on a read path makes every graph question pay for it.

Measured sweep cost:

live entries one sweep cost via rmw_get_topic_names_and_types
1335 1.4 ms 2.9 ms
9235 21.3 ms 42.6 ms
12671 27.2 ms 54.4 ms

rmw_get_topic_names_and_types calls query_all twice, once for publishers and once for subscriptions (rmw_graph.cpp#L261-L262), so it pays the full cost twice.

Anything that polls the graph turns this into a continuous load. A recorder polling the topic list at the common 10 Hz default costs about 0.5 s of CPU per second against a registry with ~9000 entries, and it pays that whether or not the graph changed.

Suggested fix

  • Rate limit registry_cleanup_stale to at most once per interval per process. One second is plenty. Small change, large effect.
  • Make rmw_get_topic_names_and_types sweep once instead of twice.
  • Longer term, drop the per-slot stat() entirely. A dead peer already makes the doorbell sendto fail with ECONNREFUSED, so liveness can be learned without a syscall per entity.

Worth noting separately: stat("/proc/<pid>") also succeeds for a zombie, since the entry survives until the parent reaps it. A slot whose owner has exited but not been reaped is therefore treated as alive and never reclaimed by this path.

Activity

  1. benaliabderrahmane commented on Aug 13, 2026

    @benaliabderrahmane
    OwnerAuthor

    Fix in flight: PR #58 (fix/throttle-registry-cleanup) caps the sweep at once per second per context on the graph-query path, while keeping the two safety-critical sweep sites (rmw_init, registry-full fallback) unthrottled. Still needed after #60: the pull-based replay removed the doorbell storm, not the per-query stat() sweep.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

P0Do nowperformanceThroughput/latency work

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions