Reducing NeMo-Speech working-set memory usage on Windows
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- Half a day
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- cpp
- Domain
- operating-systems, performance
Research direction
The payload names no source files, so start from the model loading path where the GGUF loader and mmap are mentioned, and check whether the Windows-only memory calls belong in a platform-specific file. The reporter's experiment used the Windows EmptyWorkingSet() API, so confirm first with a maintainer whether an optional trim is wanted at all. Done means an opt-in trim that runs without unloading the model, with the measured working set reported before and after.
Written by the indexing model from the issue text.
Description
Hi, I have been testing NeMo-Speech in Windows on a Nvidia GPU using CUDA. I'm RAM-limited because I have a lot of other software running which was one of the reasons I wanted to offload the speech model to the GPU.
My issue is after loading, nemo-speech.exe was showing about 633 MB of private working set in Windows Task Manager, despite the model weights being loaded onto the GPU. I wondered whether mmap or the GGUF loader's temporary buffers were responsible, but disabling mmap did not resolve the issue. Using VMMap showed substantial committed heap memory, although only a fraction of it was resident in physical RAM.
I experimented with the Windows API EmptyWorkingSet(), which allows Windows to remove pages from a process's working set without terminating the process or releasing its underlying allocations.
The results were:
Initial private working set | ~633 MB
Working set after trimming | ~2.2 MB
After a subsequent transcription | ~50 MB
During continued dictation | ~around 55MB and it settles there.
Private bytes | ~6.38 GB (unchanged)
So I seem to have recovered about 580MB of system RAM without a noticeable downside.
Would it be worth considering an optional Windows-specific working-set trimming mechanism? A one-off trim could substantially reduce the resident RAM footprint without unloading the model, and it can then grow naturally during use.
- Dominant language
- C++
- Stars
- 207
- Forks
- 47
- Avg merge
- 13h 10m
- Merged PRs (30d)
- 12
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/NeMo-Speech.cpp
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
NVIDIA/NeMo-Speech.cpp#98 ·
Maintainers usually reply within 1 day
-
Windows: CUDA Backend Fails to Initialize RTX 4050 While Vulkan Backend WorksPossibly taken @pskrunner14 claimed this 2 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 38/100
NVIDIA/NeMo-Speech.cpp#96 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
NVIDIA/NeMo-Speech.cpp#94 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
NVIDIA/NeMo-Speech.cpp#93 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 57/100
NVIDIA/NeMo-Speech.cpp#92 ·
Maintainers usually reply within 1 day
All issues in NVIDIA/NeMo-Speech.cpp
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
arduino/ArduinoCore-megaavr#152 ·
-
bug needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
microsoft/microsoft-ui-xaml#12515 · 1 comment · 1 reaction ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
FujiNetWIFI/fujinet-firmware#2261 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
mw00/project-maya#798 ·
Maintainers usually reply within 1 day