Skip to content

fix: surface EMSGSIZE as RMW_RET_ERROR on publish - #3

Merged
benaliabderrahmane merged 1 commit into
mainfrom
feature/emsgsize-error-propagation
Jun 5, 2026
Merged

benaliabderrahmane merged 1 commit into
mainfrom
feature/emsgsize-error-propagation

Conversation

@benaliabderrahmane

Copy link
Copy Markdown
Owner

Summary

rmw_publish previously discarded send_to()'s return value and always returned RMW_RET_OK. When sendmsg() failed with EMSGSIZE — datagram larger than the kernel-clamped SO_SNDBUF, e.g. because net.core.wmem_max was at the distribution default — the application saw "success" while no subscriber received anything. Silent data-loss bug at roughly 416 KB and above on a default-configured host.

Changes

  • send_to()'s bool return becomes enum class SendResult { Ok, SoftDrop, ConfigError }. EMSGSIZE → ConfigError; EAGAIN / ENOBUFS / ENOENT / ECONNREFUSED / unknown errno → SoftDrop. Existing throttled log messages are unchanged.
  • rmw_publish and rmw_publish_serialized_message propagate ConfigError as RMW_RET_ERROR with a descriptive RMW_SET_ERROR_MSG. SoftDrop semantics are preserved — applications don't see exceptions for routine slow-subscriber / peer-teardown drops on a non-blocking transport.
  • One-shot per-process check at rmw_init: warn if /proc/sys/net/core/{wmem,rmem}_max is below what we request via SO_{SND,RCV}BUF. Surfaces the misconfiguration at startup instead of on the first big publish.

Test plan

  • PublishReturnsErrorOnEMSGSIZE — shrink SO_SNDBUF on the context's send socket; assert rmw_publish_serialized_message of a 64 KB payload returns RMW_RET_ERROR.
  • PublishStillReturnsOkOnSoftDropPeerGone — unlink a sub's socket file under the publisher's cache; assert the next rmw_publish still returns RMW_RET_OK (regression guard for over-escalating soft drops).
  • Both tests fail on main and pass on this branch (verified via git stash + rebuild + ctest).
  • Full ctest suite: 12/12 pass.

rmw_publish previously discarded send_to()'s return value and always
returned RMW_RET_OK. When sendmsg() failed with EMSGSIZE (datagram
larger than the kernel-clamped SO_SNDBUF, e.g. net.core.wmem_max not
tuned), the application saw "success" while no subscriber received
anything. This was a silent data-loss bug at ~416 KB and above on a
default-configured host.

Replace send_to()'s bool return with an enum SendResult that
distinguishes ConfigError (EMSGSIZE — hard, non-transient) from
SoftDrop (EAGAIN/ENOBUFS/ENOENT/ECONNREFUSED — routine for a
non-blocking transport). rmw_publish and rmw_publish_serialized_message
now propagate ConfigError as RMW_RET_ERROR with a descriptive
RMW_SET_ERROR_MSG; SoftDrop semantics are unchanged (still logged
inside send_to, still RMW_RET_OK to the caller).

Also add a one-shot per-process check at rmw_init that warns if
/proc/sys/net/core/{wmem,rmem}_max is below the SO_{SND,RCV}BUF we
request, so the misconfiguration surfaces at startup instead of on
the first big publish.

Tests:
- PublishReturnsErrorOnEMSGSIZE: shrinks SO_SNDBUF and asserts that
  rmw_publish_serialized_message returns RMW_RET_ERROR.
- PublishStillReturnsOkOnSoftDropPeerGone: unlinks a sub's socket
  and asserts the next publish still returns RMW_RET_OK (ENOENT
  must not be propagated).

Both fail on HEAD and pass with this change.
@benaliabderrahmane benaliabderrahmane added the bug Something isn't working label Jun 1, 2026
@benaliabderrahmane

Copy link
Copy Markdown
Owner Author

Review: fix: surface EMSGSIZE as RMW_RET_ERROR on publish

Approving. This is a clean, focused, correct fix for a real silent-data-loss bug. Verified against the PR branch (feature/emsgsize-error-propagation, base main), not the diverged working tree.

What I checked:

  • Core fix is correct. send_to's bool→enum class SendResult { Ok, SoftDrop, ConfigError } cleanly separates the hard EMSGSIZE failure from transient drops. Both rmw_publish (rmw_publisher.cpp:247-260) and rmw_publish_serialized_message (~307-321) accumulate config_error over the fanout and return RMW_RET_ERROR with a descriptive RMW_SET_ERROR_MSG; SoftDrop (EAGAIN/ENOENT/ECONNREFUSED/...) stays RMW_RET_OK, preserving the non-blocking transport's drop semantics.
  • TRANSIENT_LOCAL current publish is covered. cached.payload = payload is a copy and the TRANSIENT_LOCAL block falls through to the unified checked loop at rmw_publisher.cpp:249-256, so an oversized current publish returns RMW_RET_ERROR for both VOLATILE and TRANSIENT_LOCAL.
  • No collateral breakage. The bool→enum change does not break the other send_to call sites in rmw_service.cpp / rmw_client.cpp (they discard the return as bare statements). CI is green on jazzy / kilted / rolling.
  • Tests. PublishReturnsErrorOnEMSGSIZE (escalation) and PublishStillReturnsOkOnSoftDropPeerGone (regression guard against over-escalating soft drops) both fail on main and pass here; the #include "types.hpp" addition is justified (uses UdsContext.send_socket_fd / UdsSubscription.socket_path).

Non-blocking, optional pre-merge polish:

  • The non-serialized rmw_publish EMSGSIZE branch (rmw_publisher.cpp:249-256) has no direct test — only the serialized path is asserted. The two blocks are textually identical so risk is low, but a one-line rmw_publish assertion with an oversized payload would close the gap and guard future edits.
  • The TRANSIENT_LOCAL late-joiner replay loop (rmw_publisher.cpp:237-239) still discards send_to's return. This is the pre-existing best-effort pattern from main and is fine, but a one-line comment marking replay as intentionally best-effort would prevent it reading as an oversight against the PR body's framing.
  • The startup sysctl-undersize warning (transport.cpp:324-358, wired at rmw_init.cpp:111) is a separable proactive-diagnostics feature bundled into the reactive fix. It is genuinely useful — it catches the silent SO_SNDBUF kernel clamp that the existing setsockopt warning does not — and is disclosed in the PR body, so this is documented scope rather than silent creep; ideally a separate commit. Heads-up for the fleet: on an untuned host (wmem_max/rmem_max default ~208KB < 48MB) this WARN fires once per node at boot, so consider shipping a /etc/sysctl.d/ baseline alongside.

LGTM.

@benaliabderrahmane
benaliabderrahmane merged commit dbf1097 into main Jun 5, 2026
3 checks passed
@benaliabderrahmane
benaliabderrahmane deleted the feature/emsgsize-error-propagation branch June 17, 2026 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant