Isaac ROS 5: component container crashes (SIGSEGV) on shutdown, each node library embeds its own cuda_buffer IPC manager thread
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 45/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- cpp
- Área
- backend, distributed-systems
Línea de trabajo
Start by comparing the exported CudaVmmIPCManager symbols in the listed node libraries with libcuda_buffer.so, then inspect how those libraries link cuda_buffer and how its dispatcher threads stop during unload. Reproduce the SIGINT shutdown with the component pipeline and check whether the dispatcher threads are joined or share one manager. Done means the container exits cleanly without dispatcher threads executing unmapped code.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Summary
On Isaac ROS 5, a component container that runs the RT-DETR object detection pipeline crashes with SIGSEGV (exit code -11) almost every time it is stopped with Ctrl+C. In some runs it hangs instead, and ros2 launch has to SIGKILL it after SIGINT and SIGTERM.
The core dump points to the cuda_buffer IPC manager: every Isaac ROS node library embeds its own copy of cuda_buffer_backend::CudaVmmIPCManager, and each copy runs its own FDDispatcher thread. When the container unloads the node libraries at shutdown, these threads keep running code that is no longer mapped.
Environment
- Jetson AGX Thor, JetPack 7.2 (L4T 39.2), MAXN, CUDA 13.2
- Isaac ROS 5.0.0 buildfarm packages, ROS 2 Lyrical, Docker (Isaac ROS CLI 2.6.0)
ros-lyrical-cuda-buffer0.1.2,rmw_fastrtps_cpp- Pipeline in one
component_container(--executor-type multi-threaded):ResizeNode,PadNode,ImageFormatConverterNode,ImageToTensorNode,InterleavedToPlanarNode,ReshapeNode,TensorRTNode,RtDetrPreprocessorNode,RtDetrDecoderNode, fed by a camera node in another container at 30 FPS
Results
| Nodes loaded in the container | Stop with SIGINT |
|---|---|
| Full pipeline, inference running | SIGSEGV in 11 of 12 runs, 1 to 4 s after SIGINT |
| Full pipeline, input images published as plain CPU data | SIGSEGV in 4 of 4 runs |
Full pipeline, --executor-type single-threaded |
SIGSEGV in 3 of 4 runs |
ResizeNode only |
clean exit (2 runs) |
Image and tensor nodes without TensorRTNode |
clean exit (2 runs) |
TensorRTNode only, no input |
clean exit (2 runs) |
Image and tensor nodes + TensorRTNode, inference running |
SIGSEGV in 2 of 2 runs |
Under gdb the container exits normally, so the crash depends on timing.
Core dump
The crashing thread executes an address that belongs to no loaded library:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000fffd180dbf08 in ?? ()
Other threads are in FDDispatcher::run() from different node libraries, and some of them also run code at unmapped addresses:
Thread 15: #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libresize_node.so
Thread 11: #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libinterleaved_to_planar_node.so
Thread 7: #1 cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run() () from /opt/ros/lyrical/lib/libimage_to_tensor_node.so
Thread 18: #0 __GI_epoll_pwait #1 0x0000fffd12f9e278 in ?? ()
Thread 16: #0 __GI_epoll_pwait #1 0x0000fffd1175a418 in ?? ()
Thread 14: #5 cuda_buffer_backend::CudaBuffer::BufferRecycler::run() () from /opt/ros/lyrical/lib/libcuda_buffer.so
Each node library defines the IPC manager itself instead of using libcuda_buffer.so:
$ nm -DC /opt/ros/lyrical/lib/libresize_node.so | grep -E " T .*(get_dispatcher|FDDispatcher::run)"
0000000000117b10 T cuda_buffer_backend::CudaVmmIPCManager::FDDispatcher::run()
0000000000117d10 T cuda_buffer_backend::CudaVmmIPCManager::get_dispatcher()
The same symbols are defined in libpad_node.so, libimage_format_converter_node.so, libimage_to_tensor_node.so, libinterleaved_to_planar_node.so, libreshape_node.so, libtensor_rt_node.so, librtdetr_preprocessor_node.so, librtdetr_decoder_node.so and libcuda_buffer.so.
Expected behavior
The container stops cleanly on SIGINT: the dispatcher threads are stopped and joined before their library is unloaded, or the node libraries share the single IPC manager of libcuda_buffer.so.
- Lenguaje dominante
- C++
- Estrellas
- 324
- Forks
- 229
- Merge medio
- 1 h 4 min
- PR fusionados (30 d)
- 1
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de NVIDIA-ISAAC-ROS/isaac_ros_common
-
`version_info.yaml` is written into the source tree for `ament_python` packagesPosiblemente ocupada @Daniiiil1 la tomó hace 105 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
-
Humble image: rclcpp with cherry-picked #2109 aborts action servers ("Taking data from action server but no ready event") - needs iron's take_data workaround (#2531)Posiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. Abierto
Dificultad 4/5 3-5 días Aptitud para principiantes 38/100
-
Post-quantum authentication transparency in Isaac ROS relay topologies (MAVLink + ROS2 bridge)Abierto
Dificultad 5/5 Más de una semana Aptitud para principiantes 25/100
-
Release 3.2 Dockerfile.base numpy version mismatchQuizá libre de nuevo @swapnesh-wani-nvidia la tomó hace 204 días y no hay ningún pull request abierto. Abiertobug
NVIDIA-ISAAC-ROS/isaac_ros_common#235 · 1 comentario · 1 asignado ·
-
Isaac ROS Dev Base Catalog Page says it's Humble not JazzyQuizá libre de nuevo @chengronglai la tomó hace 254 días y no hay ningún pull request abierto. Abiertodocumentation
NVIDIA-ISAAC-ROS/isaac_ros_common#232 · 1 asignado ·
Todos los issues de NVIDIA-ISAAC-ROS/isaac_ros_common
Issues similares
-
bug
Dificultad 1/5 Menos de una hora Aptitud para principiantes 85/100
Los mantenedores suelen responder en 1 día
-
external
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
ydb-platform/ydb#60347 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 88/100
BZFlag-Dev/bzflag#405 ·
-
bug: unconfirmed status: needs triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
PrismLauncher/PrismLauncher#6525 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Desync when watching replayAbiertoBug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Warzone2100/warzone2100#5163 ·
Los mantenedores suelen responder en 1 día