EventsCBGExecutor timers on ROS time stall after a backward /clock jump
Los mantenedores suelen responder en 1 día
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 30/100
Línea de trabajo
Start with the reproducer in jump_probe.cpp: build it and run ./jump_probe cbg next to single and multi. The cause is in EventsCBGExecutor's TimerQueue::running_timers, which caches deadlines that timer_thread() sleeps on, and ClockWaiter wakes that thread on each clock sample. Done means the ROS-clock timer fires about 20 times in the backward-jump phase without calling timer->reset(), and the late use_sim_time case also fires.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Generated by Generative AI
No response
Operating System:
Ubuntu 26.04
ROS version or commit hash:
rolling at bca9dffc (rclcpp 33.1.0, rcl 10.5.1); Kilted ros-kilted-rclcpp 29.5.11 behaves the same
RMW implementation (if applicable):
No response
RMW Configuration (if applicable):
No response
Client library (if applicable):
rclcpp
'ros2 doctor --report' output
ros2 doctor --report
<COPY OUTPUT HERE>
Steps to reproduce issue
Build and run ./jump_probe cbg, then compare with single and multi.
The program runs two nodes:
clock_pubpublishes/clockevery 20 ms.subjectusesuse_sim_time: truewith three 100 ms timers: ROS (create_timer), steady (create_wall_timer), and system (RCL_SYSTEM_TIME).
The clock runs for 2 s from 1000 s, jumps backward to 900 s for 2 s, then forward to 1004 s for 2 s.
Variants:
late: Enablesuse_sim_timeafter timer creation and executor startup.reset: Resets the ROS timer immediately after the backward jump.
#include <atomic>
#include <chrono>
#include <cstdio>
#include <memory>
#include <string>
#include <thread>
#include <rclcpp/executors/events_cbg_executor/events_cbg_executor.hpp>
#include <rclcpp/experimental/executors/events_executor/events_executor.hpp>
#include <rclcpp/rclcpp.hpp>
#include <rosgraph_msgs/msg/clock.hpp>
using namespace std::chrono_literals;
int main(int argc, char ** argv)
{
const std::string kind = argc > 1 ? argv[1] : "cbg";
const std::string mode = argc > 2 ? argv[2] : "-";
const bool reset_after_jump = mode == "reset";
const bool late_sim_time = mode == "late";
rclcpp::init(argc, argv);
auto clock_node = std::make_shared<rclcpp::Node>("clock_pub");
auto pub = clock_node->create_publisher<rosgraph_msgs::msg::Clock>("/clock", rclcpp::ClockQoS());
rclcpp::NodeOptions opts;
if (!late_sim_time) {
opts.parameter_overrides({rclcpp::Parameter("use_sim_time", true)});
}
auto subject = std::make_shared<rclcpp::Node>("subject", opts);
std::atomic<int> ros_fires{0}, steady_fires{0}, system_fires{0};
auto ros_timer = subject->create_timer(100ms, [&] {++ros_fires;});
auto steady_timer = subject->create_wall_timer(100ms, [&] {++steady_fires;});
auto system_clock = std::make_shared<rclcpp::Clock>(RCL_SYSTEM_TIME);
auto system_timer = rclcpp::create_timer(
subject, system_clock, rclcpp::Duration{100ms}, [&] {++system_fires;});
std::shared_ptr<rclcpp::Executor> exec;
if (kind == "single") {
exec = std::make_shared<rclcpp::executors::SingleThreadedExecutor>();
} else if (kind == "multi") {
exec = std::make_shared<rclcpp::executors::MultiThreadedExecutor>(rclcpp::ExecutorOptions{}, 2);
} else if (kind == "events") {
exec = std::make_shared<rclcpp::experimental::executors::EventsExecutor>();
} else {
exec = std::make_shared<rclcpp::executors::EventsCBGExecutor>(rclcpp::ExecutorOptions{}, 2);
}
exec->add_node(subject);
std::thread spinner{[&] {exec->spin();}};
if (late_sim_time) {
std::this_thread::sleep_for(200ms);
subject->set_parameter(rclcpp::Parameter("use_sim_time", true));
}
rosgraph_msgs::msg::Clock msg;
const auto play = [&](std::int64_t start_ns, const char * phase) {
const int r0 = ros_fires, s0 = steady_fires, y0 = system_fires;
for (int i = 0; i < 100; ++i) { // 2 s of sim time at real rate, in 20 ms steps
msg.clock = rclcpp::Time{start_ns + i * 20'000'000LL};
pub->publish(msg);
std::this_thread::sleep_for(20ms);
}
std::printf(
"%-6s %-5s %-28s ros=%2d steady=%2d system=%2d\n", kind.c_str(), mode.c_str(),
phase, ros_fires - r0, steady_fires - s0, system_fires - y0);
};
play(1000'000'000'000LL, "before jump (1000 s)");
if (reset_after_jump) {
msg.clock = rclcpp::Time{900'000'000'000LL};
pub->publish(msg);
std::this_thread::sleep_for(50ms);
ros_timer->reset();
}
play(900'000'000'000LL, "after backward jump (900 s)");
play(1004'000'000'000LL, "after forward jump (1004 s)");
exec->cancel();
spinner.join();
rclcpp::shutdown();
return 0;
}
cmake_minimum_required(VERSION 3.20)
project(jump_probe CXX)
set(CMAKE_CXX_STANDARD 20)
find_package(rclcpp REQUIRED)
find_package(rosgraph_msgs REQUIRED)
add_executable(jump_probe jump_probe.cpp)
target_link_libraries(jump_probe rclcpp::rclcpp ${rosgraph_msgs_TARGETS})
Expected behavior
The timer on the ROS clock keeps firing through the backward jump, as it does on SingleThreadedExecutor and MultiThreadedExecutor, and it starts firing once use_sim_time is enabled late.
Actual behavior
Timer callbacks per 2 s phase (20 expected), rolling at bca9dffc:
| Executor | mode | before jump (1000 s) | after backward jump (900 s) | after forward jump (1004 s) |
|---|---|---|---|---|
SingleThreadedExecutor |
- | ros=20 steady=20 system=20 | ros=19 steady=20 system=20 | ros=20 steady=20 system=20 |
MultiThreadedExecutor |
- | ros=20 steady=20 system=20 | ros=19 steady=20 system=20 | ros=20 steady=20 system=20 |
EventsCBGExecutor |
- | ros=20 steady=20 system=20 | ros=0 steady=20 system=20 | ros=20 steady=20 system=20 |
EventsCBGExecutor |
late | ros=0 steady=20 system=20 | ros=0 steady=20 system=20 | ros=0 steady=20 system=20 |
EventsCBGExecutor |
reset | ros=20 steady=20 system=20 | ros=19 steady=20 system=20 | ros=20 steady=21 system=21 |
EventsExecutor (experimental) |
- | ros=20 steady=20 system=20 | ros=19 steady=20 system=20 | ros=20 steady=20 system=20 |
EventsExecutor (experimental) |
late | ros=19 steady=20 system=20 | ros=19 steady=20 system=20 | ros=20 steady=20 system=20 |
With EventsCBGExecutor the ROS clock timer resumes only once simulated time passes the deadline it had before the jump, so with ros2 bag play --loop --clock it fires once per loop at best. In the late case the timer keeps a deadline in system time (about 1.76e9 s) and never fires for any realistic simulated time. Steady and system clock timers are unaffected on every executor, and forward jumps are fine.
Additional information
Root cause: rcl rebases timer deadlines on backward clock jumps and ROS time activation/deactivation without invoking the on-reset callback (intentional per ros2/rcl#995). EventsCBGExecutor caches these deadlines in TimerQueue::running_timers, which is only updated on timer creation, callback completion, or reset.
After a clock jump, timer_thread() keeps sleeping against the stale deadline even though ClockWaiter wakes it on every clock sample. The timer never fires because its updated deadline isn't reflected in the queue.
Calling timer->reset() re-keys the timer and restores execution. Wait-set executors and the experimental EventsExecutor are unaffected because they re-query rcl deadlines.
Workaround: Reset sim-time timers in a post-jump callback configured with a negative min_backward threshold. For late ROS time activation, create timers after enabling use_sim_time.
- Lenguaje dominante
- C++
- Estrellas
- 815
- Forks
- 572
- Merge medio
- 6 d 8 h
- PR fusionados (30 d)
- 20
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de ros2/rclcpp
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
Los mantenedores suelen responder en 1 día
-
Late /clock delivery under use_sim_time stalls node now() for ~0.33 s (found via ros2_control controller time)Posiblemente ocupada @JeremiahM37 la tomó hoy. Abierto
Dificultad 5/5 Más de una semana Aptitud para principiantes 10/100
ros2/rclcpp#3329 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
-
:farmer: Flaky EventsCBGExecutor test: reentrant callback group fails to run callbacks concurrentlyAbiertobug
Dificultad 4/5 3-5 días Aptitud para principiantes 35/100
Los mantenedores suelen responder en 1 día
-
Copy constructor exception leaks intra-process message allocationPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abierto
Dificultad 3/5 1-2 días Aptitud para principiantes 35/100
Los mantenedores suelen responder en 1 día
-
[Jazzy] action server registers clock jump callbacks without the clock mutex (heap corruption)Posiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abiertobug
Dificultad 4/5 3-5 días Aptitud para principiantes 44/100
Los mantenedores suelen responder en 1 día
Todos los issues de ros2/rclcpp
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
accessibility good first issue hacktoberfest help wanted
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
OpenChemistry/avogadrolibs#3184 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
Tracktion/tracktion_engine#475 ·
Los mantenedores suelen responder en 1 día
-
CB name typo in LMAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 62/100
orbiternassp/NASSP#1701 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
redis/memtier_benchmark#732 ·
Los mantenedores suelen responder en 1 día