No history yet

Kernel Tuning

The Real-Time Kernel

Off-the-shelf Linux schedulers are designed for fairness and throughput, not deterministic, low-latency performance. For robotics, where a missed deadline can mean a collision, this is unacceptable. The solution is the PREEMPT_RT patchset, which transforms the kernel into a fully preemptible, real-time system.

This patch minimizes the amount of kernel code that is non-preemptible, turning components like spinlocks into mutexes. The result is a dramatic reduction in scheduling latency, ensuring that high-priority robotics tasks can execute the moment they are ready, rather than waiting for a long-running kernel operation to complete.

To meet the requirements of real-time systems, an RTOS must include certain key features, such as preemptive, priority-based scheduling, a preemptive kernel, and minimizing latency.

The process involves patching your kernel source, then configuring it to enable full preemption. With ROS2 Jazzy's native support for the Raspberry Pi 5, you can tune hardware-specific parameters with greater confidence. After patching, you'll configure the kernel, typically via make menuconfig.

# Navigate to General setup --->
#   Preemption Model --->
#     (X) Fully Preemptible Kernel (Real-Time)

CPU Isolation and Affinity

A real-time kernel prevents kernel tasks from delaying your application, but what about other user-space tasks? CPU isolation, or shielding, reserves specific CPU cores for your critical ROS2 executors. The general Linux scheduler won't place any other processes on these shielded cores, eliminating jitter from context switching.

This is configured using the isolcpus kernel boot parameter in your GRUB configuration. For example, isolcpus=2,3 would shield the third and fourth cores from the scheduler. However, shielding a core is only half the battle. By default, hardware interrupts can still be processed on any core, including your shielded ones. This is where IRQ affinity comes in. You must manually reassign interrupt requests away from your real-time cores to prevent the hardware itself from interrupting your critical loops.

Once cores are isolated and interrupts are managed, you explicitly assign your ROS2 executor threads to these cores using taskset. You'll also set a real-time scheduling policy like SCHED_FIFO (First-In, First-Out) or SCHED_RR (Round-Robin). SCHED_FIFO is often preferred for control loops, as it runs a task to completion (or until it blocks or is preempted by a higher-priority task) without time-slicing.

# Assign a process with PID 1234 to core 2
# and set its scheduling policy to FIFO with priority 98
sudo chrt -f -p 98 1234
sudo taskset -p -c 2 1234

Network and Memory Tuning

ROS2's performance is heavily dependent on the underlying DDS implementation, which relies on UDP for discovery and data transport. High-frequency, high-bandwidth topics can easily overwhelm default kernel network buffer settings, leading to dropped packets and discovery failures.

Tuning the network stack via sysctl is essential. You need to increase the maximum socket buffer sizes to handle the bursty nature of DDS traffic, especially during node discovery or when publishing large sensor data like point clouds.

# /etc/sysctl.conf

# Increase max socket receive buffer size to 32MB
net.core.rmem_max=33554432

# Increase max socket send buffer size to 32MB
net.core.wmem_max=33554432

# Increase default receive buffer size
net.core.rmem_default=16777216

# Increase default send buffer size
net.core.wmem_default=16777216

Beyond networking, memory management can introduce latency. A occurs when a process tries to access a part of its memory that has been swapped to disk. This I/O operation can take milliseconds—an eternity for a real-time system. To prevent this, critical processes should use mlockall(). This system call locks the process's entire address space into RAM, ensuring that no part of its memory will ever be paged out.

With these kernel-level optimizations, you create a stable, predictable foundation for your ROS2 system, ensuring that your software's performance is determined by your algorithms, not by the whims of the OS scheduler.