EventsCBGExecutor timers on ROS time stall after a backward /clock jump
Maintainer antworten meist innerhalb von 1 Tag
Bewertung
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Anfängerfreundlichkeit
- 30/100
Rechercherichtung
Start with the reproducer in jump_probe.cpp: build it and run ./jump_probe cbg next to single and multi. The cause is in EventsCBGExecutor's TimerQueue::running_timers, which caches deadlines that timer_thread() sleeps on, and ClockWaiter wakes that thread on each clock sample. Done means the ROS-clock timer fires about 20 times in the backward-jump phase without calling timer->reset(), and the late use_sim_time case also fires.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Generated by Generative AI
No response
Operating System:
Ubuntu 26.04
ROS version or commit hash:
rolling at bca9dffc (rclcpp 33.1.0, rcl 10.5.1); Kilted ros-kilted-rclcpp 29.5.11 behaves the same
RMW implementation (if applicable):
No response
RMW Configuration (if applicable):
No response
Client library (if applicable):
rclcpp
'ros2 doctor --report' output
ros2 doctor --report
<COPY OUTPUT HERE>
Steps to reproduce issue
Build and run ./jump_probe cbg, then compare with single and multi.
The program runs two nodes:
clock_pubpublishes/clockevery 20 ms.subjectusesuse_sim_time: truewith three 100 ms timers: ROS (create_timer), steady (create_wall_timer), and system (RCL_SYSTEM_TIME).
The clock runs for 2 s from 1000 s, jumps backward to 900 s for 2 s, then forward to 1004 s for 2 s.
Variants:
late: Enablesuse_sim_timeafter timer creation and executor startup.reset: Resets the ROS timer immediately after the backward jump.
#include <atomic>
#include <chrono>
#include <cstdio>
#include <memory>
#include <string>
#include <thread>
#include <rclcpp/executors/events_cbg_executor/events_cbg_executor.hpp>
#include <rclcpp/experimental/executors/events_executor/events_executor.hpp>
#include <rclcpp/rclcpp.hpp>
#include <rosgraph_msgs/msg/clock.hpp>
using namespace std::chrono_literals;
int main(int argc, char ** argv)
{
const std::string kind = argc > 1 ? argv[1] : "cbg";
const std::string mode = argc > 2 ? argv[2] : "-";
const bool reset_after_jump = mode == "reset";
const bool late_sim_time = mode == "late";
rclcpp::init(argc, argv);
auto clock_node = std::make_shared<rclcpp::Node>("clock_pub");
auto pub = clock_node->create_publisher<rosgraph_msgs::msg::Clock>("/clock", rclcpp::ClockQoS());
rclcpp::NodeOptions opts;
if (!late_sim_time) {
opts.parameter_overrides({rclcpp::Parameter("use_sim_time", true)});
}
auto subject = std::make_shared<rclcpp::Node>("subject", opts);
std::atomic<int> ros_fires{0}, steady_fires{0}, system_fires{0};
auto ros_timer = subject->create_timer(100ms, [&] {++ros_fires;});
auto steady_timer = subject->create_wall_timer(100ms, [&] {++steady_fires;});
auto system_clock = std::make_shared<rclcpp::Clock>(RCL_SYSTEM_TIME);
auto system_timer = rclcpp::create_timer(
subject, system_clock, rclcpp::Duration{100ms}, [&] {++system_fires;});
std::shared_ptr<rclcpp::Executor> exec;
if (kind == "single") {
exec = std::make_shared<rclcpp::executors::SingleThreadedExecutor>();
} else if (kind == "multi") {
exec = std::make_shared<rclcpp::executors::MultiThreadedExecutor>(rclcpp::ExecutorOptions{}, 2);
} else if (kind == "events") {
exec = std::make_shared<rclcpp::experimental::executors::EventsExecutor>();
} else {
exec = std::make_shared<rclcpp::executors::EventsCBGExecutor>(rclcpp::ExecutorOptions{}, 2);
}
exec->add_node(subject);
std::thread spinner{[&] {exec->spin();}};
if (late_sim_time) {
std::this_thread::sleep_for(200ms);
subject->set_parameter(rclcpp::Parameter("use_sim_time", true));
}
rosgraph_msgs::msg::Clock msg;
const auto play = [&](std::int64_t start_ns, const char * phase) {
const int r0 = ros_fires, s0 = steady_fires, y0 = system_fires;
for (int i = 0; i < 100; ++i) { // 2 s of sim time at real rate, in 20 ms steps
msg.clock = rclcpp::Time{start_ns + i * 20'000'000LL};
pub->publish(msg);
std::this_thread::sleep_for(20ms);
}
std::printf(
"%-6s %-5s %-28s ros=%2d steady=%2d system=%2d\n", kind.c_str(), mode.c_str(),
phase, ros_fires - r0, steady_fires - s0, system_fires - y0);
};
play(1000'000'000'000LL, "before jump (1000 s)");
if (reset_after_jump) {
msg.clock = rclcpp::Time{900'000'000'000LL};
pub->publish(msg);
std::this_thread::sleep_for(50ms);
ros_timer->reset();
}
play(900'000'000'000LL, "after backward jump (900 s)");
play(1004'000'000'000LL, "after forward jump (1004 s)");
exec->cancel();
spinner.join();
rclcpp::shutdown();
return 0;
}
cmake_minimum_required(VERSION 3.20)
project(jump_probe CXX)
set(CMAKE_CXX_STANDARD 20)
find_package(rclcpp REQUIRED)
find_package(rosgraph_msgs REQUIRED)
add_executable(jump_probe jump_probe.cpp)
target_link_libraries(jump_probe rclcpp::rclcpp ${rosgraph_msgs_TARGETS})
Expected behavior
The timer on the ROS clock keeps firing through the backward jump, as it does on SingleThreadedExecutor and MultiThreadedExecutor, and it starts firing once use_sim_time is enabled late.
Actual behavior
Timer callbacks per 2 s phase (20 expected), rolling at bca9dffc:
| Executor | mode | before jump (1000 s) | after backward jump (900 s) | after forward jump (1004 s) |
|---|---|---|---|---|
SingleThreadedExecutor |
- | ros=20 steady=20 system=20 | ros=19 steady=20 system=20 | ros=20 steady=20 system=20 |
MultiThreadedExecutor |
- | ros=20 steady=20 system=20 | ros=19 steady=20 system=20 | ros=20 steady=20 system=20 |
EventsCBGExecutor |
- | ros=20 steady=20 system=20 | ros=0 steady=20 system=20 | ros=20 steady=20 system=20 |
EventsCBGExecutor |
late | ros=0 steady=20 system=20 | ros=0 steady=20 system=20 | ros=0 steady=20 system=20 |
EventsCBGExecutor |
reset | ros=20 steady=20 system=20 | ros=19 steady=20 system=20 | ros=20 steady=21 system=21 |
EventsExecutor (experimental) |
- | ros=20 steady=20 system=20 | ros=19 steady=20 system=20 | ros=20 steady=20 system=20 |
EventsExecutor (experimental) |
late | ros=19 steady=20 system=20 | ros=19 steady=20 system=20 | ros=20 steady=20 system=20 |
With EventsCBGExecutor the ROS clock timer resumes only once simulated time passes the deadline it had before the jump, so with ros2 bag play --loop --clock it fires once per loop at best. In the late case the timer keeps a deadline in system time (about 1.76e9 s) and never fires for any realistic simulated time. Steady and system clock timers are unaffected on every executor, and forward jumps are fine.
Additional information
Root cause: rcl rebases timer deadlines on backward clock jumps and ROS time activation/deactivation without invoking the on-reset callback (intentional per ros2/rcl#995). EventsCBGExecutor caches these deadlines in TimerQueue::running_timers, which is only updated on timer creation, callback completion, or reset.
After a clock jump, timer_thread() keeps sleeping against the stale deadline even though ClockWaiter wakes it on every clock sample. The timer never fires because its updated deadline isn't reflected in the queue.
Calling timer->reset() re-keys the timer and restores execution. Wait-set executors and the experimental EventsExecutor are unaffected because they re-query rcl deadlines.
Workaround: Reset sim-time timers in a post-jump callback configured with a negative min_backward threshold. For late ROS time activation, create timers after enabling use_sim_time.
- Vorherrschende Sprache
- C++
- Sterne
- 815
- Forks
- 572
- Ø Merge
- 6 T. 8 Std.
- Gemergte PRs (30 T.)
- 20
Entwicklungsumgebung
- Kein Dockerfile und keine Docker-Compose-Datei
- Keine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus ros2/rclcpp
-
enhancement
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 70/100
Maintainer antworten meist innerhalb von 1 Tag
-
Late /clock delivery under use_sim_time stalls node now() for ~0.33 s (found via ros2_control controller time)Evtl. vergeben @JeremiahM37 hat das vor 1 Tag übernommen. Offen
Schwierigkeit 5/5 Über eine Woche Anfängerfreundlichkeit 10/100
ros2/rclcpp#3329 · 2 Kommentare ·
Maintainer antworten meist innerhalb von 1 Tag
-
:farmer: Flaky EventsCBGExecutor test: reentrant callback group fails to run callbacks concurrentlyOffenbug
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 35/100
Maintainer antworten meist innerhalb von 1 Tag
-
Copy constructor exception leaks intra-process message allocationEvtl. vergeben Ein verknüpfter Pull Request ist offen oder bereits gemergt. Offen
Schwierigkeit 3/5 1-2 Tage Anfängerfreundlichkeit 35/100
Maintainer antworten meist innerhalb von 1 Tag
-
[Jazzy] action server registers clock jump callbacks without the clock mutex (heap corruption)Evtl. vergeben Ein verknüpfter Pull Request ist offen oder bereits gemergt. Offenbug
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 44/100
Maintainer antworten meist innerhalb von 1 Tag
Ähnliche Issues
-
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 85/100
521xueweihan/HelloGitHub#4011 ·
-
docs(simulation): Quickstart publishes UDP 14550, which the SITL images' own docs say not to doOffenstatus:needs-triage
Schwierigkeit 1/5 1-3 Stunden Anfängerfreundlichkeit 78/100
PX4/PX4-Autopilot#29292 · 1 Kommentar ·
Maintainer antworten meist innerhalb von 1 Tag
-
I/O internal correctness medium priority rt_lib
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 79/100
Maintainer antworten meist innerhalb von 1 Tag
-
bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 82/100
Maintainer antworten meist innerhalb von 1 Tag
-
Mac mini M6 (16 GB): Q2_0 streaming works - measured 11.7->27.2 tok/s + arena fix for sub-32 GB MacsOffen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
Maintainer antworten meist innerhalb von 1 Tag