Skip to content

Commit d1c8e19

Browse files
benaliabderrahmanebenaliabderrahmane
committed
temp
1 parent 7d961ee commit d1c8e19

1 file changed

Lines changed: 156 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# rmw_unix_socket_cpp
2+
3+
A lightweight, **localhost-only** RMW (ROS Middleware) implementation for ROS 2,
4+
built on Unix domain sockets and POSIX shared memory instead of DDS.
5+
6+
It targets single-machine deployments with many nodes (150+ in production) and a
7+
small, deterministic resource footprint — no DDS, no daemon, no background
8+
threads. Discovery is a shared-memory registry; data moves over `AF_UNIX`
9+
datagram sockets; the wait set is `epoll` + `eventfd`.
10+
11+
## When to use it (and when not)
12+
13+
**Use it when:**
14+
- Everything runs on **one Linux host** (bare metal or co-located containers).
15+
- You want lower overhead and more predictable resource usage than DDS.
16+
- You don't need network communication, security, or DDS interoperability.
17+
18+
**Don't use it when:**
19+
- Nodes span **multiple machines** — `AF_UNIX` is local-only. Use CycloneDDS or Zenoh.
20+
- You need DDS interop, content filtering, zero-copy loaned messages, or
21+
deadline/lifespan/liveliness QoS. See [Limitations](#limitations).
22+
23+
## Status
24+
25+
- ROS 2 **Jazzy**, **Kilted**, and **Rolling** — built and tested in CI.
26+
- **Linux / x86_64 only** (uses `epoll`, `eventfd`, `/dev/shm`, `AF_UNIX`).
27+
- Version `0.1.0`, Apache-2.0.
28+
29+
## Quick start
30+
31+
```bash
32+
# Build (from your workspace root, alongside your other packages)
33+
source /opt/ros/jazzy/setup.bash
34+
colcon build --packages-select rmw_unix_socket_cpp
35+
36+
# Select it at runtime — every process that should talk must set this
37+
source install/setup.bash
38+
export RMW_IMPLEMENTATION=rmw_unix_socket_cpp
39+
40+
# Verify
41+
ros2 run demo_nodes_cpp talker
42+
# ...in another terminal (with the same two env exports):
43+
ros2 run demo_nodes_cpp listener
44+
ros2 doctor --report # confirms the active middleware
45+
```
46+
47+
Without `RMW_IMPLEMENTATION=rmw_unix_socket_cpp`, rcl loads whatever RMW is the
48+
system default (usually FastDDS) and this implementation is never used. **All**
49+
nodes that need to communicate must select the same RMW — there is no DDS bridge.
50+
51+
## Design at a glance
52+
53+
| Concern | Choice |
54+
|---|---|
55+
| **Transport** | `SOCK_DGRAM` Unix domain sockets in `/tmp/ros2_uds/<domain_id>/`. Message boundaries are free; no connection management. |
56+
| **Discovery** | Single shared-memory file `/dev/shm/ros2_uds_<domain_id>`. No daemon, no multicast. |
57+
| **Locking** | Lock-free registry (per-slot seqlock + atomic state machine); process-local caches use a plain mutex. |
58+
| **Wait** | `epoll` over socket fds + `eventfd` guard conditions. **No background threads** — all I/O happens inside `rmw_wait`. |
59+
| **Serialization** | CDR via `fastcdr` + `rosidl_typesupport_fastrtps` (the same path as the DDS RMWs). |
60+
| **Stale cleanup** | Dead processes are detected via `/proc/<pid>`; their registry entries and socket files are reclaimed on init and on every discovery query. |
61+
62+
Every choice — including the ones that were tried and abandoned — is written up
63+
with rationale in **[DESIGN.md](rmw_unix_socket_cpp/DESIGN.md)**.
64+
65+
## Running across multiple containers
66+
67+
Because discovery and cleanup live in shared kernel resources (`/dev/shm`,
68+
`/tmp`, and the PID namespace), co-located containers must **share** those
69+
namespaces to see each other. Each container also needs `RMW_IMPLEMENTATION` set.
70+
71+
Required per container:
72+
73+
| Requirement | Why |
74+
|---|---|
75+
| `RMW_IMPLEMENTATION=rmw_unix_socket_cpp` | Selects this middleware. |
76+
| `--ipc=host` | All containers share `/dev/shm` → the same discovery registry. |
77+
| `--pid=host` | Stale-PID cleanup uses `/proc/<pid>`; without a shared PID namespace one container reclaims another's live entries and corrupts the registry. |
78+
| `-v /tmp/ros2_uds:/tmp/ros2_uds` | All containers share the socket files data actually flows over. |
79+
| same `ROS_DOMAIN_ID` | The registry file and socket directory are per-domain. |
80+
81+
`docker compose` example:
82+
83+
```yaml
84+
services:
85+
talker:
86+
image: my_ros_image
87+
environment:
88+
- RMW_IMPLEMENTATION=rmw_unix_socket_cpp
89+
- ROS_DOMAIN_ID=0
90+
ipc: host
91+
pid: host
92+
volumes:
93+
- /tmp/ros2_uds:/tmp/ros2_uds
94+
command: ros2 run demo_nodes_cpp talker
95+
96+
listener:
97+
image: my_ros_image
98+
environment:
99+
- RMW_IMPLEMENTATION=rmw_unix_socket_cpp
100+
- ROS_DOMAIN_ID=0
101+
ipc: host
102+
pid: host
103+
volumes:
104+
- /tmp/ros2_uds:/tmp/ros2_uds
105+
command: ros2 run demo_nodes_cpp listener
106+
```
107+
108+
`docker run` equivalent:
109+
110+
```bash
111+
docker run --ipc=host --pid=host \
112+
-e RMW_IMPLEMENTATION=rmw_unix_socket_cpp -e ROS_DOMAIN_ID=0 \
113+
-v /tmp/ros2_uds:/tmp/ros2_uds \
114+
my_ros_image ros2 run demo_nodes_cpp talker
115+
```
116+
117+
### Host kernel tuning
118+
119+
Set these on the **host** (they apply to all containers). Without them you get
120+
silent buffer clamps and transient drops under load — fine for small control
121+
messages, but images and point clouds will be dropped.
122+
123+
| Knob | Default | Recommended | Why |
124+
|---|--:|--:|---|
125+
| `net.core.rmem_max` | ~212 KB | 48 MB+ | Caps `SO_RCVBUF`; large messages drop below this. |
126+
| `net.core.wmem_max` | ~212 KB | 48 MB+ | Caps `SO_SNDBUF`; `EMSGSIZE` on send below this. |
127+
| `net.unix.max_dgram_qlen` | 512 | 4096 | Per-socket pending-datagram cap; bursty fan-out overflows 512. |
128+
| `ulimit -n` (`nofile`) | 1024 | 65536 | Each pub/sub/service/client uses 1–2 fds. |
129+
130+
```bash
131+
sudo sysctl -w net.core.rmem_max=48000000 net.core.wmem_max=48000000 \
132+
net.unix.max_dgram_qlen=4096
133+
```
134+
135+
See [DESIGN.md → Operational requirements](rmw_unix_socket_cpp/DESIGN.md#operational-requirements)
136+
for the full rationale.
137+
138+
## Limitations
139+
140+
Localhost only; Linux only; no DDS interoperability. Loaned (zero-copy) messages,
141+
dynamic message types, network flow endpoints, and content filtering return
142+
`RMW_RET_UNSUPPORTED`. `deadline`, `lifespan`, and `liveliness` QoS are accepted
143+
but not enforced. The full list and the reasoning behind each is in
144+
[DESIGN.md → Limitations and Unsupported Features](rmw_unix_socket_cpp/DESIGN.md#limitations-and-unsupported-features).
145+
146+
## Documentation
147+
148+
| Document | What it is |
149+
|---|---|
150+
| **[DESIGN.md](rmw_unix_socket_cpp/DESIGN.md)** | Self-contained design rationale: every wire format, sync primitive, QoS decision, and operational quirk. Start here. |
151+
| **[test/README.md](rmw_unix_socket_cpp/test/README.md)** | Every test in the suite, what it validates, and why it exists. |
152+
| **[test/perf/README.md](rmw_unix_socket_cpp/test/perf/README.md)** | Standalone latency/scale benchmarks (not run by `colcon test`). |
153+
154+
## License
155+
156+
Apache-2.0.

0 commit comments

Comments
 (0)