ROS 2 Shared Memory, Measured: Fast DDS SHM and Data-Sharing vs Cyclone DDS + iceoryx

Why I measured this

I had two old notes on ROS 2 shared memory. One was a Fast DDS XML profile with data-sharing turned on. The other was a “use Cyclone DDS with iceoryx” recipe. Both promised lower latency for big messages. Neither had a number in it.

So I built a small pub/sub pair, ran it on ROS 2 Jazzy in Docker, and measured six transport setups at 1 KB, 1 MB and 4 MB.

The setup

• Host: Intel i7-12700H (20 threads), Linux 7.0, Docker 29.9, everything in one container per run.
• ROS: ros:jazzy-ros-base, rmw_fastrtps_cpp 8.4.4 on Fast DDS 2.14.7, rmw_cyclonedds_cpp 2.2.4 on Cyclone DDS 0.10.5 with iceoryx 2.0.6.
• Traffic: one publisher process, one subscriber process, 100 Hz, reliable QoS, KEEP_LAST(16). 50 warm-up messages, then 500 measured ones. Each case was run once.
• Latency: the publisher writes steady_clock nanoseconds into the message right before publish(). The subscriber reads the clock first thing in its callback. Both processes share the same monotonic clock, so the difference is end to end: serialization, transport, the executor waking up, deserialization.

There are four message types. Three are fixed size, so they’re plain memory that can be loaned:

# Fixed1K.msg / Fixed1M.msg / Fixed4M.msg
uint64 stamp_ns
uint8[1024] data        # 1048576 / 4194304 in the other two

The fourth is unbounded, like sensor_msgs/Image:

# Dyn.msg
uint64 stamp_ns
uint8[] data

There are two publish modes:

• copy: publish(const Msg&). This is what most code does.
• loan: borrow_loaned_message(), write the stamp, then publish(std::move(loaned)). One catch for honesty: in loan mode I don’t write the 4 MB payload, only the stamp. A real camera driver would fill the buffer, so treat the loan numbers as the transport floor.

The six setups:

The first trap: Jazzy disables subscriber loans

My first Cyclone + iceoryx run looked like nothing changed. 4 MB took about 3 ms either way. The publisher reported can_loan_messages() == true, but the subscription said false.

I traced through rmw_cyclonedds and Cyclone’s reader code. The iceoryx subscriber was being created fine. The “no” came from rcl. Jazzy turns subscriber-side loans off unless you opt in, and rclcpp tells you so once you set the variable:

[WARN] [bench_sub]: Loaned messages are only safe with const ref subscription callbacks.
If you are using any other kind of subscriptions, set the ROS_DISABLE_LOANED_MESSAGES
environment variable to 1 (the default).

So by default, a loaned publish still lands in a freshly allocated message on the subscriber side, and every byte gets copied. To get real zero-copy on Jazzy you need both of these:

export ROS_DISABLE_LOANED_MESSAGES=0

node->create_subscription<Msg>("bench", qos, [&](const Msg & m) { /* ... */ });

The callback has to take a const Msg&. A SharedPtr or by-value callback isn’t safe with loans. Here’s the 4 MB fixed message on Cyclone + iceoryx, measured before I knew this:

That was a 50-message smoke test. The full runs below all use loan mode with ROS_DISABLE_LOANED_MESSAGES=0. Copy mode leaves it at the default.

Results

4 MB, median latency

The bars use a log scale, one # per 1/12 of a decade above 10 µs. The numbers are p50 in µs.

fastdds-udp          fixed copy  |#####################################     12066
fastdds-udp          fixed loan  |#####################################     11093
fastdds-udp          dyn copy    |#####################################     12165
fastdds-default      fixed copy  |##############################             2936
fastdds-default      fixed loan  |####################################      10601
fastdds-default      dyn copy    |#############################              2652
fastdds-shm64        fixed copy  |###########################                1621
fastdds-shm64        fixed loan  |#####################                       521
fastdds-shm64        dyn copy    |##########################                 1407
fastdds-datasharing  fixed copy  |########################                   1058
fastdds-datasharing  fixed loan  |##############                              147
fastdds-datasharing  dyn copy    |#####################################     12177
cyclone-udp          fixed copy  |##############################             3087
cyclone-udp          fixed loan  |#############################              2419
cyclone-udp          dyn copy    |#############################              2476
cyclone-iox          fixed copy  |###############################            3535
cyclone-iox          fixed loan  |###########                                  83
cyclone-iox          dyn copy    |##############################             3348

cyclone-udp can’t loan, so its “loan” row is just a second copy run.

Full table

The loan column shows which side actually got a loan: pub+sub, only pub, or no. All times are in µs, n = 500 for every row except one 4 MB fastdds-default loan run, which got 498.

What the numbers say

At 1 KB, nothing matters. Every setup sits between 59 and 192 µs median. That’s executor wake-up and scheduling, not the transport.

iceoryx only pays off with loans on both ends. cyclone-iox in copy mode is as slow as plain UDP at 4 MB: 3.5 ms against 3.1 ms. Turn on loans and the median drops to 83 µs, flat across 1 MB and 4 MB. That flat line is the whole point of zero-copy. The size stops mattering.

Fast DDS out of the box falls over above 512 KB. With no XML, the 1 MB and 4 MB tails jump to 11, 21, 31 ms and beyond. The 4 MB loan median is 10.6 ms. The jumps come in steps of about 10 ms, which looks like reliable-protocol recovery, not slow copying. The Fast DDS 2.14 docs list the default SHM segment_size as 512 KB. With the segment raised to 64 MB, the steps went away: 4 MB loan p99 dropped from 70.9 ms to 1.1 ms.

<transport_descriptor>
    <transport_id>shm_big</transport_id>
    <type>SHM</type>
    <segment_size>67108864</segment_size>
</transport_descriptor>

UDP-only Fast DDS is the worst case for big messages. It sits around 11–12 ms median at 1 MB and up, with the same 10 ms step pattern.

Data-sharing is fast, and my old profile was a trap. The profile from my note turns off the builtin transports and keeps only UDPv4, then turns on data-sharing:

<participant profile_name="participant_profile_ros2" is_default_profile="true">
    <rtps>
        <userTransports><transport_id>UDP_transport</transport_id></userTransports>
        <useBuiltinTransports>false</useBuiltinTransports>
    </rtps>
</participant>
<data_writer profile_name="default publisher profile" is_default_profile="true">
    <qos>
        <publishMode><kind>ASYNCHRONOUS</kind></publishMode>
        <data_sharing><kind>AUTOMATIC</kind></data_sharing>
    </qos>
    <historyMemoryPolicy>PREALLOCATED_WITH_REALLOC</historyMemoryPolicy>
</data_writer>

For fixed-size types it works well: 147 µs median at 4 MB with loans, and 1.06 ms in copy mode, the best copy number in the table. But data-sharing only covers bounded types. An unbounded message like sensor_msgs/Image falls back to the only transport left, UDP, and goes to 12 ms median with a 44 ms p99. If you use a profile like this, keep SHM in userTransports next to UDP.

Data-sharing only showed up with the XML profile. I listed /dev/shm during each 4 MB loan run:

fastdds-default:     fastrtps*, fastrtps_portN*, sem.fastrtps_portN_mutex   (SHM transport only)
fastdds-shm64:       fastrtps*, fastrtps_portN*, sem.fastrtps_portN_mutex   (SHM transport only)
fastdds-datasharing: fast_datasharing_01.0f.…_0.0.14.3, fast_datasharing_…_0.0.14.4
cyclone-iox:         iceoryx_mgmt, root

Without the profile and RMW_FASTRTPS_USE_QOS_FROM_XML=1, I never saw a fast_datasharing_* segment. The publisher still reported it could loan, but the bytes went through the SHM transport. That’s still a copy into the segment.

Small things that cost me time

• Check the profile variable name. Fast DDS 2.14 in Jazzy only knows FASTRTPS_DEFAULT_PROFILES_FILE. I checked with strings libfastrtps.so | grep PROFILES_FILE. Look before you copy a variable name from a newer guide.
• RouDi needs room. With my mempool config (512×2 KB, 64×1 MB, 64×4 MB), RouDi reserved 65.7 MB for iceoryx_mgmt and 336.7 MB for the data segment. Docker’s default /dev/shm is 64 MB, so I ran with --shm-size=2g. Across containers you’d use --ipc=host instead, so they share one /dev/shm.
• RouDi pools must fit your biggest message, including the chunk header. I sized the pools at a bit over 1 MB and 4 MB.
• The Cyclone config that worked with 0.10.5:

<CycloneDDS xmlns="https://cdds.io/config">
    <Domain id="any">
        <SharedMemory>
            <Enable>true</Enable>
            <LogLevel>warn</LogLevel>
        </SharedMemory>
    </Domain>
</CycloneDDS>

iox-roudi -c roudi.toml &
export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
export CYCLONEDDS_URI=file://$PWD/config/cyclonedds_shm.xml
export ROS_DISABLE_LOANED_MESSAGES=0

What I’d do now

For big fixed-size messages on one host: Cyclone DDS + iceoryx, loaned publish, const& callbacks, and ROS_DISABLE_LOANED_MESSAGES=0. That was the only setup where 4 MB cost the same as 1 KB.

If you’re staying on Fast DDS, raise the SHM segment_size first. That one change fixed the tails. Then add data-sharing for the bounded types, without dropping the SHM transport.

For unbounded types like images and point clouds, no setup here gets you zero-copy. The best I measured was fastdds-shm64 at 1.4 ms median for 4 MB.

Caveats

• One machine, one run per case, 500 samples, 100 Hz. Your scheduler and core count will move these numbers. The 10 ms steps and the loan-vs-copy gaps were large enough that I don’t expect them to flip.
• In loan mode the publisher doesn’t fill the payload.
• Both processes ran in the same container. That’s same-host IPC, not intra-process.

Reproduce

The package is a plain ament_cmake package with rosidl_generate_interfaces for the four .msg files. One gotcha: project(shm_bench C CXX). The generated C typesupport needs the C language enabled, or CMake fails with “Cannot determine link language”.

FROM ros:jazzy-ros-base
RUN apt-get update && apt-get install -y --no-install-recommends ros-jazzy-rmw-cyclonedds-cpp \
    && rm -rf /var/lib/apt/lists/*
COPY ws /ws
RUN . /opt/ros/jazzy/setup.sh && cd /ws && colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release

One case looks like this. Start the subscriber, then the publisher:

docker run --rm --shm-size=2g -v $PWD/config:/cfg:ro shm-bench:jazzy bash -c '
  . /ws/install/setup.bash
  export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp CYCLONEDDS_URI=file:///cfg/cyclonedds_shm.xml
  export ROS_DISABLE_LOANED_MESSAGES=0
  iox-roudi -c /cfg/roudi.toml & sleep 2
  B=/ws/install/shm_bench/lib/shm_bench/bench
  $B sub 4m 0 loan 500 & S=$!; sleep 1
  $B pub 4m 0 loan 500; wait $S'

The RouDi mempool config I used:

[general]
version = 1

[[segment]]

[[segment.mempool]]
size = 2048
count = 512

[[segment.mempool]]
size = 1049600
count = 64

[[segment.mempool]]
size = 4195328
count = 64