Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Memory leak in exploration_node causes severe slowdown during full HM3D v2 benchmark (~14 GB RSS, 30+ min/episode after ~400 eps)

Aperta
  • Qualcosa non va?
#41 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Tranquilla
Stack tecnologico
cpp, python
Ambito
performance, robotics

Direzione di ricerca

Start with exploration_fsm.cpp lines 14-20 and trace the episode-reset calls from exploration_node. Reproduce the HM3D v2 val run with habitat_evaluation.py and monitor RSS plus the per-episode times in videos/test_hm3dv2_val/record.txt. Done means memory remains bounded across episodes without restarting the planner and slowdown no longer accumulates.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

While running the full HM3D v2 val benchmark (1000 episodes) with habitat_evaluation.py, I observed severe slowdown over time caused by what appears to be a memory leak in the C++ exploration_node. After ~420 episodes the planner's RSS grew to ~14 GB and per-episode time went from ~37s (episode 0) to 30+ minutes (episode ~420). Restarting only the ROS 2 planner (while keeping VLM servers and habitat_evaluation.py's continue.txt checkpoint) brought per-episode time back to ~60s and RSS down to ~500 MB.

Environment

  • OS: Ubuntu 22.04 (jammy) — note: ROS 2 Jazzy is Tier 3 here, source-built per official docs
  • ROS 2: Jazzy (built from source against the conda env's Python)
  • conda env: apexnav (Python 3.9, per apexnav_environment.yaml)
  • Branch: ros2-jazzy
  • GPU: NVIDIA RTX 4090, CUDA 12.8, PyTorch 2.8.0+cu128
  • VLM servers (default ports 12181-12184): all running fresh, never restarted during the run
  • Command: 
    python habitat_evaluation.py --dataset hm3dv2 \
    habitat.dataset.split=val \
    need_video=true \
    test_epi_num=-1

Symptoms

Episode # Per-episode time exploration_node RSS
1 37 s ~500 MB (fresh)
100 ~70 s ~2 GB
300 ~150 s ~6 GB
420 ~30 min ~13.9 GB

After episode ~420 the system began swapping (60 GB total RAM, planner alone holding ~14 GB), turning the 60-second exploration loops into 30+ minutes due to OS-level page thrashing.

Reproduction

  1. Run the full HM3D v2 val split with test_epi_num=-1 (1000 episodes) and need_video=true.
  2. Periodically observe RSS of exploration_node via: 
    ps -o pid,rss,etime,cmd -p $(pgrep -f exploration_node)
  3. Check the cumulative seconds spend in this task field in videos/test_hm3dv2_val/record.txt — the per-episode delta grows roughly monotonically from ~40s to thousands of seconds.

Workaround (works, but obviously not ideal)

I wrote a tmux watchdog that restarts only the ROS 2 planner every 200 episodes (or whenever its RSS exceeds 5 GB), while keeping the 4 VLM servers and habitat_evaluation.py running. Because continue.txt checkpoints per-episode, evaluation resumes seamlessly from where it left off, and per-episode time returns to ~30-60s after each restart.

Possible Causes

In exploration_fsm.cpp lines 14-20 the FSM code does fp_.reset(...), fd_.reset(...), expl_manager_.reset(...) etc. on each episode reset, which suggests inter-episode reset was intended. But something downstream — possibly:

  • a shared_ptr to point clouds / occupancy grids / SDF map is held by another component (visualization? mapping? planner cache?) and never released
  • PCL / Octomap-style data structures that grow but never shrink
  • accumulated frontiers / TF buffer / parameter listeners

—keeps the previous episode's allocations alive across resets.

Thanks for the great work.

Lingua principale
C
Stelle
462
Fork
38
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Robotics-STAR-Lab/ApexNav

Tutte le issue di Robotics-STAR-Lab/ApexNav

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.