Isaac ROS 5: component container crashes (SIGSEGV) on shutdown, each node library embeds its own cuda_buffer IPC manager thread
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 45/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- cpp
- Ambito
- backend, distributed-systems
Direzione di ricerca
Start by comparing the exported CudaVmmIPCManager symbols in the listed node libraries with libcuda_buffer.so, then inspect how those libraries link cuda_buffer and how its dispatcher threads stop during unload. Reproduce the SIGINT shutdown with the component pipeline and check whether the dispatcher threads are joined or share one manager. Done means the container exits cleanly without dispatcher threads executing unmapped code.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
On Isaac ROS 5, a component container that runs the RT-DETR object detection pipeline crashes with SIGSEGV (exit code -11) almost every time it is stopped with Ctrl+C. In some runs it hangs instead, and ros2 launch has to SIGKILL it after SIGINT and SIGTERM.
The core dump points to the cuda_buffer IPC manager: every Isaac ROS node library embeds its own copy of cuda_buffer_backend::CudaVmmIPCManager, and each copy runs its own FDDispatcher thread. When the container unloads the node libraries at shutdown, these threads keep running code that is no longer mapped.
Environment
- Jetson AGX Thor, JetPack 7.2 (L4T 39.2), MAXN, CUDA 13.2
- Isaac ROS 5.0.0 buildfarm packages, ROS 2 Lyrical, Docker (Isaac ROS CLI 2.6.0)
ros-lyrical-cuda-buffer0.1.2,rmw_fastrtps_cpp- Pipeline in one
component_container(--executor-type multi-threaded):ResizeNode,PadNode,ImageFormatConverterNode,ImageToTensorNode,InterleavedToPlanarNode,ReshapeNode,TensorRTNode,RtDetrPreprocessorNode,RtDetrDecoderNode, fed by a camera node in another container at 30 FPS
Results
| Nodes loaded in the container | Stop with SIGINT |
|---|---|
| Full pipeline, inference running | SIGSEGV in 11 of 12 runs, 1 to 4 s after SIGINT |
| Full pipeline, input images published as plain CPU data | SIGSEGV in 4 of 4 runs |
Full pipeline, --executor-type single-threaded |
SIGSEGV in 3 of 4 runs |
ResizeNode only |
clean exit (2 runs) |
Image and tensor nodes without TensorRTNode |
clean exit (2 runs) |
TensorRTNode only, no input |
clean exit (2 runs) |
Image and tensor nodes + TensorRTNode, inference running |
SIGSEGV in 2 of 2 runs |
Under gdb the container exits normally, so the crash depends on timing.
Core dump
The crashing thread executes an address that belongs to no loaded library:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000fffd180dbf08 in ?? ()
Other threads are in FDDispatcher::run() from different node libraries, and some of them also run code at unmapped addresses:
Thread 15: #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libresize_node.so
Thread 11: #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libinterleaved_to_planar_node.so
Thread 7: #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libimage_to_tensor_node.so
Thread 18: #0 __GI_epoll_pwait #1 0x0000fffd12f9e278 in ?? ()
Thread 16: #0 __GI_epoll_pwait #1 0x0000fffd1175a418 in ?? ()
Thread 14: #5 cuda_buffer_backend::CudaBuffer::BufferRecycler::run() () from /opt/ros/lyrical/lib/libcuda_buffer.so
Each node library defines the IPC manager itself instead of using libcuda_buffer.so:
$ nm -DC /opt/ros/lyrical/lib/libresize_node.so | grep -E " T .*(get_dispatcher|FDDispatcher::run)"
0000000000117b10 T cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run()
0000000000117d10 T cuda_buffer_backend::CudaVmmIPCManager::get_dispatcher()
The same symbols are defined in libpad_node.so, libimage_format_converter_node.so, libimage_to_tensor_node.so, libinterleaved_to_planar_node.so, libreshape_node.so, libtensor_rt_node.so, librtdetr_preprocessor_node.so, librtdetr_decoder_node.so and libcuda_buffer.so.
Expected behavior
The container stops cleanly on SIGINT: the dispatcher threads are stopped and joined before their library is unloaded, or the node libraries share the single IPC manager of libcuda_buffer.so.
- Lingua principale
- C++
- Stelle
- 324
- Fork
- 229
- Merge medio
- 1h 4m
- PR unite (30g)
- 1
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di NVIDIA-ISAAC-ROS/isaac_ros_common
-
`version_info.yaml` is written into the source tree for `ament_python` packagesForse già presa @Daniiiil1 l’ha presa 105 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 86/100
-
Humble image: rclcpp with cherry-picked #2109 aborts action servers ("Taking data from action server but no ready event") - needs iron's take_data workaround (#2531)Forse già presa Una pull request collegata a questa issue è aperta o già unita. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 38/100
-
Post-quantum authentication transparency in Isaac ROS relay topologies (MAVLink + ROS2 bridge)Aperta
Difficoltà 5/5 Più di una settimana Idoneità per principianti 25/100
-
Release 3.2 Dockerfile.base numpy version mismatchForse di nuovo libera @swapnesh-wani-nvidia l’ha presa 204 giorni fa e non c’è nessuna pull request aperta. Apertabug
NVIDIA-ISAAC-ROS/isaac_ros_common#235 · 1 commento · 1 assegnatario ·
-
Isaac ROS Dev Base Catalog Page says it's Humble not JazzyForse di nuovo libera @chengronglai l’ha presa 254 giorni fa e non c’è nessuna pull request aperta. Apertadocumentation
NVIDIA-ISAAC-ROS/isaac_ros_common#232 · 1 assegnatario ·
Tutte le issue di NVIDIA-ISAAC-ROS/isaac_ros_common
Issue simili
-
bug sev-3
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
-
bug priority:p2
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 82/100
xifan2333/fcitx5-vinput#738 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
Tracktion/tracktion_engine#498 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
aristocratos/btop#1985 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
I maintainer di solito rispondono entro 1 giorno