Isaac ROS 5: component container crashes (SIGSEGV) on shutdown, each node library embeds its own cuda_buffer IPC manager thread

Aperta
#246 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp

Direzione di ricerca

Start by comparing the exported CudaVmmIPCManager symbols in the listed node libraries with libcuda_buffer.so, then inspect how those libraries link cuda_buffer and how its dispatcher threads stop during unload. Reproduce the SIGINT shutdown with the component pipeline and check whether the dispatcher threads are joined or share one manager. Done means the container exits cleanly without dispatcher threads executing unmapped code.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

On Isaac ROS 5, a component container that runs the RT-DETR object detection pipeline crashes with SIGSEGV (exit code -11) almost every time it is stopped with Ctrl+C. In some runs it hangs instead, and ros2 launch has to SIGKILL it after SIGINT and SIGTERM.

The core dump points to the cuda_buffer IPC manager: every Isaac ROS node library embeds its own copy of cuda_buffer_backend::CudaVmmIPCManager, and each copy runs its own FDDispatcher thread. When the container unloads the node libraries at shutdown, these threads keep running code that is no longer mapped.

Environment
  • Jetson AGX Thor, JetPack 7.2 (L4T 39.2), MAXN, CUDA 13.2
  • Isaac ROS 5.0.0 buildfarm packages, ROS 2 Lyrical, Docker (Isaac ROS CLI 2.6.0)
  • ros-lyrical-cuda-buffer 0.1.2, rmw_fastrtps_cpp
  • Pipeline in one component_container (--executor-type multi-threaded): ResizeNode, PadNode, ImageFormatConverterNode, ImageToTensorNode, InterleavedToPlanarNode, ReshapeNode, TensorRTNode, RtDetrPreprocessorNode, RtDetrDecoderNode, fed by a camera node in another container at 30 FPS
Results
Nodes loaded in the container Stop with SIGINT
Full pipeline, inference running SIGSEGV in 11 of 12 runs, 1 to 4 s after SIGINT
Full pipeline, input images published as plain CPU data SIGSEGV in 4 of 4 runs
Full pipeline, --executor-type single-threaded SIGSEGV in 3 of 4 runs
ResizeNode only clean exit (2 runs)
Image and tensor nodes without TensorRTNode clean exit (2 runs)
TensorRTNode only, no input clean exit (2 runs)
Image and tensor nodes + TensorRTNode, inference running SIGSEGV in 2 of 2 runs

Under gdb the container exits normally, so the crash depends on timing.

Core dump

The crashing thread executes an address that belongs to no loaded library:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000fffd180dbf08 in ?? ()

Other threads are in FDDispatcher::run() from different node libraries, and some of them also run code at unmapped addresses:

Thread 15: #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libresize_node.so
Thread 11: #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libinterleaved_to_planar_node.so
Thread 7:  #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libimage_to_tensor_node.so
Thread 18: #0 __GI_epoll_pwait  #1 0x0000fffd12f9e278 in ?? ()
Thread 16: #0 __GI_epoll_pwait  #1 0x0000fffd1175a418 in ?? ()
Thread 14: #5 cuda_buffer_backend::CudaBuffer::BufferRecycler::run() () from /opt/ros/lyrical/lib/libcuda_buffer.so

Each node library defines the IPC manager itself instead of using libcuda_buffer.so:

$ nm -DC /opt/ros/lyrical/lib/libresize_node.so | grep -E " T .*(get_dispatcher|FDDispatcher::run)"
0000000000117b10 T cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run()
0000000000117d10 T cuda_buffer_backend::CudaVmmIPCManager::get_dispatcher()

The same symbols are defined in libpad_node.so, libimage_format_converter_node.so, libimage_to_tensor_node.so, libinterleaved_to_planar_node.so, libreshape_node.so, libtensor_rt_node.so, librtdetr_preprocessor_node.so, librtdetr_decoder_node.so and libcuda_buffer.so.

Expected behavior

The container stops cleanly on SIGINT: the dispatcher threads are stopped and joined before their library is unloaded, or the node libraries share the single IPC manager of libcuda_buffer.so.

Lingua principale
C++
Stelle
324
Fork
229
Merge medio
1h 4m
PR unite (30g)
1

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA-ISAAC-ROS/isaac_ros_common

Tutte le issue di NVIDIA-ISAAC-ROS/isaac_ros_common

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.