Isaac ROS 5: component container crashes (SIGSEGV) on shutdown, each node library embeds its own cuda_buffer IPC manager thread

Abierto
#246 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
cpp

Línea de trabajo

Start by comparing the exported CudaVmmIPCManager symbols in the listed node libraries with libcuda_buffer.so, then inspect how those libraries link cuda_buffer and how its dispatcher threads stop during unload. Reproduce the SIGINT shutdown with the component pipeline and check whether the dispatcher threads are joined or share one manager. Done means the container exits cleanly without dispatcher threads executing unmapped code.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

On Isaac ROS 5, a component container that runs the RT-DETR object detection pipeline crashes with SIGSEGV (exit code -11) almost every time it is stopped with Ctrl+C. In some runs it hangs instead, and ros2 launch has to SIGKILL it after SIGINT and SIGTERM.

The core dump points to the cuda_buffer IPC manager: every Isaac ROS node library embeds its own copy of cuda_buffer_backend::CudaVmmIPCManager, and each copy runs its own FDDispatcher thread. When the container unloads the node libraries at shutdown, these threads keep running code that is no longer mapped.

Environment
  • Jetson AGX Thor, JetPack 7.2 (L4T 39.2), MAXN, CUDA 13.2
  • Isaac ROS 5.0.0 buildfarm packages, ROS 2 Lyrical, Docker (Isaac ROS CLI 2.6.0)
  • ros-lyrical-cuda-buffer 0.1.2, rmw_fastrtps_cpp
  • Pipeline in one component_container (--executor-type multi-threaded): ResizeNode, PadNode, ImageFormatConverterNode, ImageToTensorNode, InterleavedToPlanarNode, ReshapeNode, TensorRTNode, RtDetrPreprocessorNode, RtDetrDecoderNode, fed by a camera node in another container at 30 FPS
Results
Nodes loaded in the container Stop with SIGINT
Full pipeline, inference running SIGSEGV in 11 of 12 runs, 1 to 4 s after SIGINT
Full pipeline, input images published as plain CPU data SIGSEGV in 4 of 4 runs
Full pipeline, --executor-type single-threaded SIGSEGV in 3 of 4 runs
ResizeNode only clean exit (2 runs)
Image and tensor nodes without TensorRTNode clean exit (2 runs)
TensorRTNode only, no input clean exit (2 runs)
Image and tensor nodes + TensorRTNode, inference running SIGSEGV in 2 of 2 runs

Under gdb the container exits normally, so the crash depends on timing.

Core dump

The crashing thread executes an address that belongs to no loaded library:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000fffd180dbf08 in ?? ()

Other threads are in FDDispatcher::run() from different node libraries, and some of them also run code at unmapped addresses:

Thread 15: #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libresize_node.so
Thread 11: #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libinterleaved_to_planar_node.so
Thread 7:  #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libimage_to_tensor_node.so
Thread 18: #0 __GI_epoll_pwait  #1 0x0000fffd12f9e278 in ?? ()
Thread 16: #0 __GI_epoll_pwait  #1 0x0000fffd1175a418 in ?? ()
Thread 14: #5 cuda_buffer_backend::CudaBuffer::BufferRecycler::run() () from /opt/ros/lyrical/lib/libcuda_buffer.so

Each node library defines the IPC manager itself instead of using libcuda_buffer.so:

$ nm -DC /opt/ros/lyrical/lib/libresize_node.so | grep -E " T .*(get_dispatcher|FDDispatcher::run)"
0000000000117b10 T cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run()
0000000000117d10 T cuda_buffer_backend::CudaVmmIPCManager::get_dispatcher()

The same symbols are defined in libpad_node.so, libimage_format_converter_node.so, libimage_to_tensor_node.so, libinterleaved_to_planar_node.so, libreshape_node.so, libtensor_rt_node.so, librtdetr_preprocessor_node.so, librtdetr_decoder_node.so and libcuda_buffer.so.

Expected behavior

The container stops cleanly on SIGINT: the dispatcher threads are stopped and joined before their library is unloaded, or the node libraries share the single IPC manager of libcuda_buffer.so.

Lenguaje dominante
C++
Estrellas
324
Forks
229
Merge medio
1 h 4 min
PR fusionados (30 d)
1

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA-ISAAC-ROS/isaac_ros_common

Todos los issues de NVIDIA-ISAAC-ROS/isaac_ros_common

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.