EventsCBGExecutor keeps queued events of removed nodes, component container segfaults on shutdown
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 18/100
Direzione di ricerca
Work is already claimed: open PR #3313. The crash is in EventsCBGExecutor vs component_container_events_cbg shutdown: CBGScheduler::remove_callback_group() is never called (FIXME in EventsCBGExecutor::sync_callback_groups() in scheduler.hpp), so queued ReadyEntity weak_ptrs outlive unloaded component libraries. A gist reproducer is linked; swapping executor vs ComponentManager destruction order is noted as a workaround. Do not start a competing patch.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Generated by Generative AI
Claude Code (claude-opus-5-5): root-cause analysis, reproducer and fix were developed with it; I reviewed and verified the results.
Operating System:
Ubuntu 24.04.5 LTS, x86_64
ROS version or commit hash:
Jazzy, ros-jazzy-rclcpp 28.1.22; also present on rolling (ebf57e35)
RMW implementation (if applicable):
rmw_zenoh_cpp and rmw_fastrtps_cpp (both crash 10/10 with the reproducer)
RMW Configuration (if applicable):
No response
Client library (if applicable):
rclcpp
'ros2 doctor --report' output
No response
Steps to reproduce issue
A component container with the EventsCBGExecutor crashes on Ctrl-C if a loaded component still has events queued in the executor when it shuts down.
Full reproducer (ament package, build and run instructions):
https://gist.github.com/spurnvoj/368641954e673feea50e89dd2e007074
Two components in separate libraries, loaded with intra-process comms into the stock component_container_events_cbg:
// libpub.so: publishes nav_msgs/Odometry every 2 ms
timer_ = create_wall_timer(2ms, [this] {
pub_->publish(std::make_unique<nav_msgs::msg::Odometry>());
});
// libsub.so: 20 subscriptions with a slow callback, so events stay queued
for (int i = 0; i < 20; i++) {
subs_.push_back(create_subscription<nav_msgs::msg::Odometry>(
"odom", 10, [](nav_msgs::msg::Odometry::ConstSharedPtr) {
std::this_thread::sleep_for(2ms);
}));
}
- Build both libraries with
-fno-gnu-unique. GCC otherwise marks some symbolsSTB_GNU_UNIQUEand glibc never really unloads the library, which hides the crash. Packages from the ROS build farm have no such symbols, so they unload. - Load both with
use_intra_process_comms: trueintocomponent_container_events_cbg. - After a few seconds, send SIGINT (Ctrl-C) to the launch process group.
Result: 10 of 10 runs crash. Without the slow callback (nothing left in the queue) it crashes only occasionally (1 of 20).
Expected behavior
The container shuts down cleanly.
Actual behavior
The container exits with SIGSEGV (exit code -11) in the executor destructor:
#0 0x...839a in ?? () from /opt/ros/jazzy/lib/librclcpp.so
#1 0x...ac53 in ?? () from /opt/ros/jazzy/lib/librclcpp.so
#2 0x...ae46 in ?? () from /opt/ros/jazzy/lib/librclcpp.so
#3 rclcpp::executors::EventsCBGExecutor::~EventsCBGExecutor() () from /opt/ros/jazzy/lib/librclcpp.so
#4 0x... in ?? ()
#5 main ()
The faulting instruction is a virtual call (call *0x18(%rax), the control block's _M_destroy) through a vtable at an unmapped address. libsub.so and libpub.so are no longer mapped in the core dump.
Additional information
Root cause: the scheduler never forgets callback groups. CBGScheduler::remove_callback_group() (scheduler.hpp) is never called (see // FIXME inform scheduler about remove cbgs in EventsCBGExecutor::sync_callback_groups()). So every CallbackGroupHandle stays alive together with its ready_entities queue until the executor is destroyed. Queued but not executed events are ReadyEntitys holding weak_ptrs to the node's entities.
In component_container_events_cbg the executor is created before the ComponentManager and therefore destroyed after it. ~ComponentManager destroys the components and unloads their libraries. ~EventsCBGExecutor then releases the last weak_ptrs, and freeing the control block calls code that lived in the unloaded library.
Swapping the declaration order in the container's main() (executor destroyed first) avoids it: 0 of 5 crashes with the reproducer (CONTAINER=workaround in the gist).
Not fixed by #3190 (28.1.22 includes it). Related: #2567 (same symptom, different cause).
In our system (MRS UAV system, several component containers) this crashed the core container in 4 of 15 Ctrl-C runs.
I have a fix that clears the queued events when the callback group's entity cache is destroyed, with a regression test. PR to follow.
- Lingua principale
- C++
- Stelle
- 814
- Fork
- 572
- Merge medio
- 4g 8h
- PR unite (30g)
- 18
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di ros2/rclcpp
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 10/100
I maintainer di solito rispondono entro 1 giorno
-
:farmer: Flaky EventsCBGExecutor test: reentrant callback group fails to run callbacks concurrentlyApertabug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 35/100
I maintainer di solito rispondono entro 1 giorno
-
Copy constructor exception leaks intra-process message allocationForse già presa @TinyGeek l’ha presa 3 giorni fa. Aperta
Difficoltà 3/5 1-2 giorni Idoneità per principianti 35/100
I maintainer di solito rispondono entro 1 giorno
-
[Jazzy] action server registers clock jump callbacks without the clock mutex (heap corruption)Forse già presa Una pull request collegata a questa issue è aperta o già unita. Apertabug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 44/100
I maintainer di solito rispondono entro 1 giorno
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
I maintainer di solito rispondono entro 1 giorno
-
`IO.FS.realPath` reports every failure as `noFileOrDirectory` instead of classifying by `errno`Apertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
leanprover/lean4#15683 ·
I maintainer di solito rispondono entro 1 giorno
-
llvm.memcpy with destination align but no source align asserts "Missed Source alignment!"Forse già presa @pvelesko l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
KhronosGroup/SPIRV-LLVM-Translator#4093 ·
I maintainer di solito rispondono entro 2 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
I maintainer di solito rispondono entro 1 giorno
-
SIMQ silently truncates multi-query batch search to the first queryForse già presa @LHT129 l’ha presa oggi. Apertakind/bug module/index needs-triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
antgroup/vsag#3080 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno