Real-time control in ROS and ROS 2.
0
Jackie Kay
jackie@osrfoundation.org
Adolfo Rodriguez Tsouroukdissian
adolfo.rodriguez@pal-robotics.com
Table of Contents
A motivating example
Real-time computing
Requirements and best practices
ROS 2 design
Comparison with ROS 1 and ros_control
Demo and results
A motivating example
A motivating example
A motivating example
Blocks can be composed by other blocks
Some blocks are subject to real-time constraints
A motivating example
Blocks can be composed by other blocks
Some blocks are subject to real-time constraints
A motivating example
Blocks can be composed by other blocks
Some blocks are subject to real-time constraints
System topology can change at runtime
7
A motivating example
Blocks can be composed by other blocks
Some blocks are subject to real-time constraints
System topology can change at runtime
8
A motivating example
Blocks can be composed by other blocks
Some blocks are subject to real-time constraints
System topology can change at runtime
9
A motivating example
Blocks can be composed by other blocks
Some blocks are subject to real-time constraints
System topology can change at runtime
10
Table of Contents
A motivating example
Real-time computing
Requirements and best practices
ROS 2 design
Comparison with ROS 1 and ros_control
Demo and results
11
Real-time computing
It's about determinism, not performance
Correct computation delivered at the correct time
Failure to respond is as bad as a wrong response
12
Real-time computing
13
Real-time computing
14
Real-time computing
Usefulness of results after missing a deadline?
15
Real-time computing
Hard real-time systems
Missing a deadline is considered a system failure
Overruns may lead to loss of life or financial damage
Safety- or mission-critical systems
Examples: reactor, aircraft and spacecraft control
16
Real-time computing
Soft real-time systems
Missing a deadline has a cost, but is not catastrophic
Result becomes less useful after deadline
Often related to Quality of Service
Examples: audio / video streaming and playback
17
Real-time computing
Firm real-time systems
Missing a deadline has a cost, but is not catastrophic
Result becomes useless after deadline
Cost might be interpreted as loss of revenue
Examples: Financial forecasting, robot assembly lines
18
Real-time computing
Why do we care?
Event response
e.g. parts inspection
Closed-loop control
e.g. manipulator control
Added benefit: Reliability, extended uptime
Downtime is unacceptable or too expensive
The above is prevalent in robotics software
19
Goal of ROS 2
Real-time compatibility, from day one
20
Table of Contents
A motivating example
Real-time computing
Requirements and best practices
ROS 2 design
Comparison with ROS 1 and ros_control
Demo and results
21
Requirements and best practices
Use an OS able to deliver the required determinism
Linux variants
OS
real-time
max latency (s)
Linux
no
104
RT PREEMPT
soft
101-102
Xenomai
hard
101
Proprietary: e.g. QNX, VxWorks
POSIX compliant, certified to IEC 61508 SIL3 et.al.
22
Requirements and best practices
Prioritize real-time threads
Use a real-time scheduling policy
23
Requirements and best practices
Prioritize real-time threads
Use a real-time scheduling policy
24
Requirements and best practices
Avoid sources of non-determinism in real-time code
Memory allocation and management ( malloc, new )
Pre-allocate resources in the non real-time path
Real-time safe O(1) allocators exist
Blocking synchronization primitives (e.g. mutex)
Real-time safe alternatives exist (e.g. lock-free)
Printing, logging ( printf, cout )
Real-time safe alternatives exist
25
Requirements and best practices
Avoid sources of non-determinism in real-time code
Network access, especially TCP/IP
RTnet stack, real-time friendly protocols like RTPS
Non real-time device drivers
Real-time drivers exist for some devices
Accessing the hard disk
Page faults
Lock address space (mlockall), pre-fault stack
26
Table of Contents
A motivating example
Real-time computing
Requirements and best practices
ROS 2 design
Comparison with ROS 1 and ros_control
Demo and results
27
ROS2 design - architecture comparison
usercode.cpp
usercode.py
usercode.cpp
usercode.c
usercode.py
roscpp
rospy
rclcpp
rclc
rclpy
UDPROS
TCPROS
ROS 2 Middleware API
Opensplice
FreeRTPS
etc.
28
ROS2 design - real-time architecture
usercode.cpp
ros_control Orocos
roscpp
UDPROS
usercode.cpp
usercode.c
rclcpp
rclc
ROS 2 Middleware API
TCPROS
Real-time Operating System
Opensplice
FreeRTPS
etc.
Real-time Operating System
29
ROS2 design Modularity
ROS2 allows customization for real-time use-cases
Memory management
Synchronization
Scheduling
are orthogonal to each other, and to node topology
30
ROS 2 - current implementation
Executor
initialization
preallocate memory
...
spin
rmw_wait(timeout)
{
pass conditions to waitset
wait (in DDS)
wake-up if timed-out
}
do work if it came in
cleanup
deallocate memory
...
non real-time
real-time
loop until interrupted
non real-time
31
ROS2 design Node lifecycle
Standard node lifecycle state machine
Opt-in feature
Node lifecycle can be managed without knowledge
of internals (black box)
Best practice from existing frameworks
microblx
OpenRTM
Orocos RTT
ros_control
32
ROS2 design Node lifecycle
credit: Geoffrey Biggs et.al.
WIP, design subject to change
33
ROS2 design Node lifecycle
credit: Geoffrey Biggs et.al.
WIP, design subject to change
34
ROS2 design Node lifecycle
credit: Geoffrey Biggs et.al.
WIP, design subject to change
35
ROS2 design Node lifecycle
Benefits of managed lifecycle
Clear separation of real-time code path
Greater control of ROS network
Help ensure correct launch sequence
Online node restart / replace
Better monitoring and supervision
Standard lifecycle standard tooling
36
ROS2 design Node composition
37
ROS2 design Node composition
Composite node is a black box with well-defined API
Lifecycle can be stepped in sync for all internal nodes
Resources can be shared for internal nodes
38
ROS2 design Communications
Inter-process
DDS can deliver soft real-time comms
Customizable QoS, can be tuned for real-time use-case
Intra-process
Efficient (zero-copy) shared pointer transport
Same-thread
No need for synchronization primitives. Simple, fast
39
ROS 2 alpha release
Real-time safety is configurable
Can configure custom allocation policy that
preallocates resources
Requires hard limit on number of pubs, subs, services
Requires messages to be statically sized
40
ROS2 progress overview
In progress
Component lifecycle
Composable components
Complete intra-process pipeline
Future work
Pre-allocate dynamic messages
CI for verifying real-time constraints
Lock-free multi-threaded executor
41
Table of Contents
A motivating example
Real-time computing
Requirements and best practices
ROS 2 design
Comparison with ROS 1 + ros_control
Demo and results
42
Comparison with ROS1 + ros_control
Real-time safe communications
Lifecycle management
Composability
43
Comparison with ROS1 + ros_control
Real-time safe communications
Lifecycle management
Composability
44
Comparison with ROS1 + ros_control
ROS1 + ros_control:
ROS2 equivalent:
drop non-standard lifecycle / interfaces gentler learning curve
smaller codebase
easier to maintain
45
Table of Contents
A motivating example
Real-time computing
Requirements and best practices
ROS 2 design
Comparison with ROS 1 and ros_control
Demo and results
46
ROS 2 Real-time Benchmarking: Setup
real-time process
motor command
controller
simulator
sensor feedback
user command
teleop
non real-time process
profiling results
logger
non real-time process
47
ROS 2 Real-time Benchmarking: Setup
Configuration
RT_PREEMPT kernel
Round robin scheduler (SCHED_RR), thread priority: 98
malloc_hook: control malloc calls
getrusage: count pagefaults
Goal
1 kHz update loop (1 ms period)
Less than 3% jitter (30 s)
Code
ros2/demos - pendulum_control
48
ROS 2 Real-time Benchmarking: Memory
Zero runtime allocations
static void * testing_malloc( size_t size, const void * caller) {
if (running) {
throw std::runtime_error("Called malloc from real-time context!" );
}
// ... allocate and return pointer...
}
Zero major pagefaults during runtime
Some minor pagefaults on the first iteration of the loop, none after
Conclusion: all required pages allocated before execution starts
49
ROS 2 Real-time Benchmarking: Results
Latency (ns)
% of update rate
Min
1620
0.16%
Max
35094
3.51%
Mean
4567
0.46%
No stress
1,070,650 cycles observed
Timeseries
Jitter histogram
50
ROS 2 Real-time Benchmarking: Results
Latency (ns)
Stress applied:
stress --cpu 2 --io 2
7,345,125 cycles observed
3 instances of overrun observed
% of update rate
Min
1398
0.14%
Max
258064
25.8%
Mean
3729.11
0.38%
Jitter histogram
51
Closing remarks
Systems subject to real-time constraints are very
relevant in robotics
ROS2 will allow user to implement such systems
with a proper RTOS, and carefully written user code
Initial results based on ROS2 alpha are encouraging
inverted pendulum demo
Design discussions and development are ongoing!
ROS SIG Next-Generation ROS
ros2 Github organization
52
Selected references
[Biggs, G.] ROS2 design article on node lifecycle (under review)
[Bruyninckx, H.] Real Time and Embedded Guide
[Kay, J.] ROS2 design article on Real-time programming
[National Instruments] What is a Real-Time Operating System (RTOS)?
[OMG] OMG RTC Specification
[ROS Control] ROS Control, an Overview
[RTT] Orocos RTT component builder's manual
[RT PREEMPT] Real-Time Linux Wiki
[Xenomai] Xenomai knowledge base
53
