Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Memory leak in exploration_node causes severe slowdown during full HM3D v2 benchmark (~14 GB RSS, 30+ min/episode after ~400 eps)

Abierto
  • ¿Algo no está bien?
#41 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Necesita aclaración
Estado de actividad
Tranquilo
Stack tecnológico
cpp, python
Área
performance, robotics

Línea de trabajo

Start with exploration_fsm.cpp lines 14-20 and trace the episode-reset calls from exploration_node. Reproduce the HM3D v2 val run with habitat_evaluation.py and monitor RSS plus the per-episode times in videos/test_hm3dv2_val/record.txt. Done means memory remains bounded across episodes without restarting the planner and slowdown no longer accumulates.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

While running the full HM3D v2 val benchmark (1000 episodes) with habitat_evaluation.py, I observed severe slowdown over time caused by what appears to be a memory leak in the C++ exploration_node. After ~420 episodes the planner's RSS grew to ~14 GB and per-episode time went from ~37s (episode 0) to 30+ minutes (episode ~420). Restarting only the ROS 2 planner (while keeping VLM servers and habitat_evaluation.py's continue.txt checkpoint) brought per-episode time back to ~60s and RSS down to ~500 MB.

Environment

  • OS: Ubuntu 22.04 (jammy) — note: ROS 2 Jazzy is Tier 3 here, source-built per official docs
  • ROS 2: Jazzy (built from source against the conda env's Python)
  • conda env: apexnav (Python 3.9, per apexnav_environment.yaml)
  • Branch: ros2-jazzy
  • GPU: NVIDIA RTX 4090, CUDA 12.8, PyTorch 2.8.0+cu128
  • VLM servers (default ports 12181-12184): all running fresh, never restarted during the run
  • Command: 
    python habitat_evaluation.py --dataset hm3dv2 \
    habitat.dataset.split=val \
    need_video=true \
    test_epi_num=-1

Symptoms

Episode # Per-episode time exploration_node RSS
1 37 s ~500 MB (fresh)
100 ~70 s ~2 GB
300 ~150 s ~6 GB
420 ~30 min ~13.9 GB

After episode ~420 the system began swapping (60 GB total RAM, planner alone holding ~14 GB), turning the 60-second exploration loops into 30+ minutes due to OS-level page thrashing.

Reproduction

  1. Run the full HM3D v2 val split with test_epi_num=-1 (1000 episodes) and need_video=true.
  2. Periodically observe RSS of exploration_node via: 
    ps -o pid,rss,etime,cmd -p $(pgrep -f exploration_node)
  3. Check the cumulative seconds spend in this task field in videos/test_hm3dv2_val/record.txt — the per-episode delta grows roughly monotonically from ~40s to thousands of seconds.

Workaround (works, but obviously not ideal)

I wrote a tmux watchdog that restarts only the ROS 2 planner every 200 episodes (or whenever its RSS exceeds 5 GB), while keeping the 4 VLM servers and habitat_evaluation.py running. Because continue.txt checkpoints per-episode, evaluation resumes seamlessly from where it left off, and per-episode time returns to ~30-60s after each restart.

Possible Causes

In exploration_fsm.cpp lines 14-20 the FSM code does fp_.reset(...), fd_.reset(...), expl_manager_.reset(...) etc. on each episode reset, which suggests inter-episode reset was intended. But something downstream — possibly:

  • a shared_ptr to point clouds / occupancy grids / SDF map is held by another component (visualization? mapping? planner cache?) and never released
  • PCL / Octomap-style data structures that grow but never shrink
  • accumulated frontiers / TF buffer / parameter listeners

—keeps the previous episode's allocations alive across resets.

Thanks for the great work.

Lenguaje dominante
C
Estrellas
472
Forks
39
Sin PR fusionados en 30 d

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Robotics-STAR-Lab/ApexNav

Todos los issues de Robotics-STAR-Lab/ApexNav

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.