EventsCBGExecutor keeps queued events of removed nodes, component container segfaults on shutdown
Los mantenedores suelen responder en 1 día
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 18/100
Línea de trabajo
Work is already claimed: open PR #3313. The crash is in EventsCBGExecutor vs component_container_events_cbg shutdown: CBGScheduler::remove_callback_group() is never called (FIXME in EventsCBGExecutor::sync_callback_groups() in scheduler.hpp), so queued ReadyEntity weak_ptrs outlive unloaded component libraries. A gist reproducer is linked; swapping executor vs ComponentManager destruction order is noted as a workaround. Do not start a competing patch.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Generated by Generative AI
Claude Code (claude-opus-5-5): root-cause analysis, reproducer and fix were developed with it; I reviewed and verified the results.
Operating System:
Ubuntu 24.04.5 LTS, x86_64
ROS version or commit hash:
Jazzy, ros-jazzy-rclcpp 28.1.22; also present on rolling (ebf57e35)
RMW implementation (if applicable):
rmw_zenoh_cpp and rmw_fastrtps_cpp (both crash 10/10 with the reproducer)
RMW Configuration (if applicable):
No response
Client library (if applicable):
rclcpp
'ros2 doctor --report' output
No response
Steps to reproduce issue
A component container with the EventsCBGExecutor crashes on Ctrl-C if a loaded component still has events queued in the executor when it shuts down.
Full reproducer (ament package, build and run instructions):
https://gist.github.com/spurnvoj/368641954e673feea50e89dd2e007074
Two components in separate libraries, loaded with intra-process comms into the stock component_container_events_cbg:
// libpub.so: publishes nav_msgs/Odometry every 2 ms
timer_ = create_wall_timer(2ms, [this] {
pub_->publish(std::make_unique<nav_msgs::msg::Odometry>());
});
// libsub.so: 20 subscriptions with a slow callback, so events stay queued
for (int i = 0; i < 20; i++) {
subs_.push_back(create_subscription<nav_msgs::msg::Odometry>(
"odom", 10, [](nav_msgs::msg::Odometry::ConstSharedPtr) {
std::this_thread::sleep_for(2ms);
}));
}
- Build both libraries with
-fno-gnu-unique. GCC otherwise marks some symbolsSTB_GNU_UNIQUEand glibc never really unloads the library, which hides the crash. Packages from the ROS build farm have no such symbols, so they unload. - Load both with
use_intra_process_comms: trueintocomponent_container_events_cbg. - After a few seconds, send SIGINT (Ctrl-C) to the launch process group.
Result: 10 of 10 runs crash. Without the slow callback (nothing left in the queue) it crashes only occasionally (1 of 20).
Expected behavior
The container shuts down cleanly.
Actual behavior
The container exits with SIGSEGV (exit code -11) in the executor destructor:
#0 0x...839a in ?? () from /opt/ros/jazzy/lib/librclcpp.so
#1 0x...ac53 in ?? () from /opt/ros/jazzy/lib/librclcpp.so
#2 0x...ae46 in ?? () from /opt/ros/jazzy/lib/librclcpp.so
#3 rclcpp::executors::EventsCBGExecutor::~EventsCBGExecutor() () from /opt/ros/jazzy/lib/librclcpp.so
#4 0x... in ?? ()
#5 main ()
The faulting instruction is a virtual call (call *0x18(%rax), the control block's _M_destroy) through a vtable at an unmapped address. libsub.so and libpub.so are no longer mapped in the core dump.
Additional information
Root cause: the scheduler never forgets callback groups. CBGScheduler::remove_callback_group() (scheduler.hpp) is never called (see // FIXME inform scheduler about remove cbgs in EventsCBGExecutor::sync_callback_groups()). So every CallbackGroupHandle stays alive together with its ready_entities queue until the executor is destroyed. Queued but not executed events are ReadyEntitys holding weak_ptrs to the node's entities.
In component_container_events_cbg the executor is created before the ComponentManager and therefore destroyed after it. ~ComponentManager destroys the components and unloads their libraries. ~EventsCBGExecutor then releases the last weak_ptrs, and freeing the control block calls code that lived in the unloaded library.
Swapping the declaration order in the container's main() (executor destroyed first) avoids it: 0 of 5 crashes with the reproducer (CONTAINER=workaround in the gist).
Not fixed by #3190 (28.1.22 includes it). Related: #2567 (same symptom, different cause).
In our system (MRS UAV system, several component containers) this crashed the core container in 4 of 15 Ctrl-C runs.
I have a fix that clears the queued events when the callback group's entity cache is destroyed, with a regression test. PR to follow.
- Lenguaje dominante
- C++
- Estrellas
- 815
- Forks
- 572
- Merge medio
- 6 d 8 h
- PR fusionados (30 d)
- 20
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de ros2/rclcpp
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
Los mantenedores suelen responder en 1 día
-
EventsCBGExecutor timers on ROS time stall after a backward /clock jumpPosiblemente ocupada @Yadunund la tomó hace 2 días. Abiertobug
Dificultad 4/5 3-5 días Aptitud para principiantes 30/100
Los mantenedores suelen responder en 1 día
-
Late /clock delivery under use_sim_time stalls node now() for ~0.33 s (found via ros2_control controller time)Posiblemente ocupada @JeremiahM37 la tomó hace 1 día. Abierto
Dificultad 5/5 Más de una semana Aptitud para principiantes 10/100
ros2/rclcpp#3329 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
-
:farmer: Flaky EventsCBGExecutor test: reentrant callback group fails to run callbacks concurrentlyAbiertobug
Dificultad 4/5 3-5 días Aptitud para principiantes 35/100
Los mantenedores suelen responder en 1 día
-
Copy constructor exception leaks intra-process message allocationPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abierto
Dificultad 3/5 1-2 días Aptitud para principiantes 35/100
Los mantenedores suelen responder en 1 día
Todos los issues de ros2/rclcpp
Issues similares
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 1 día
-
Mac mini M6 (16 GB): Q2_0 streaming works - measured 11.7->27.2 tok/s + arena fix for sub-32 GB MacsAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
Los mantenedores suelen responder en 1 día
-
champion good first issue remake source/qq
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
metasequoiaime/msime-windows#1025 ·
Los mantenedores suelen responder en 1 día
-
backlog bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
EchoTools/nevr-runtime#653 ·
Los mantenedores suelen responder en 1 día
-
`--fail-require-skip` never fires: run_tests.py searches stdout, but the unittest binary writes the skip summary to stderrPosiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día