Reducing NeMo-Speech working-set memory usage on Windows

Abierto
  • ¿Algo no está bien?
#101 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
Medio día
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp
Área
operating-systems, performance

Línea de trabajo

The payload names no source files, so start from the model loading path where the GGUF loader and mmap are mentioned, and check whether the Windows-only memory calls belong in a platform-specific file. The reporter's experiment used the Windows EmptyWorkingSet() API, so confirm first with a maintainer whether an optional trim is wanted at all. Done means an opt-in trim that runs without unloading the model, with the measured working set reported before and after.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Hi, I have been testing NeMo-Speech in Windows on a Nvidia GPU using CUDA. I'm RAM-limited because I have a lot of other software running which was one of the reasons I wanted to offload the speech model to the GPU.

My issue is after loading, nemo-speech.exe was showing about 633 MB of private working set in Windows Task Manager, despite the model weights being loaded onto the GPU. I wondered whether mmap or the GGUF loader's temporary buffers were responsible, but disabling mmap did not resolve the issue. Using VMMap showed substantial committed heap memory, although only a fraction of it was resident in physical RAM.

I experimented with the Windows API EmptyWorkingSet(), which allows Windows to remove pages from a process's working set without terminating the process or releasing its underlying allocations.

The results were:
Initial private working set | ~633 MB
Working set after trimming | ~2.2 MB
After a subsequent transcription | ~50 MB
During continued dictation | ~around 55MB and it settles there.
Private bytes | ~6.38 GB (unchanged)

So I seem to have recovered about 580MB of system RAM without a noticeable downside.

Would it be worth considering an optional Windows-specific working-set trimming mechanism? A one-off trim could substantially reduce the resident RAM footprint without unloading the model, and it can then grow naturally during use.

Lenguaje dominante
C++
Estrellas
207
Forks
47
Merge medio
13 h 10 min
PR fusionados (30 d)
12

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/NeMo-Speech.cpp

Todos los issues de NVIDIA/NeMo-Speech.cpp

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.