Observing GPU memory and/or CPU OS memory leaks with `use_persistent_mapping` enabled in `gdrdrv` during multi-process termination
@realarnavgoel is already working on this.
Since Jan 10, 2025.
Assessment
This issue has not been assessed yet.
Description
Impacted platform
All server side products, first observed on Grace-Hopper based system
Impacted gdrcopy versions
2.4.1, 2.4.2, 2.4.3
Impacted gdrcopy configs
gdrdrv driver loaded with module parameter set use_persistent_mapping=1
Scenarios
If gdrcopy persistent mapping mode is enabled,
-
If one process opens a connection to the driver (via
gdr_open), and intents to expliclity share connection (using UNIX Domain socket) with one or more processes to use the underlying connection, then the cleanup of the driver resources (viagdr_close) may be executed by one of the non-owning processes, which would be silently ignored therefore leading to CPU and GPU memory leaks. -
If a parent process A forks one or more child process B (instead of linux
fork+exec), then connections opened by A can be attempted to be closed by B during an ungraceful termination of processes via signals (SIGSEVorSIGKILL), resulting in OS and GPU memory leaks.
By default, if persistent mode is disabled, under both scenarios, GPU resources cleanup is performed through an independent workflow in CUDA driver and hence dropping the request to close this connection is benign.
Irrespective of persistent mode, this bug may lead to small CPU kernel memory leaks.
Signature of the defect
- On coherent platforms, e.g. Grace Hopper systems, GPU memory leaks can lead to unexpected side effects. For example, turning off the
nvidia-persistencedservice may hang, requiring rebooting the machine. - On non-coherent platforms, GPU memory leaks may reduce the functionality or performance of CUDA applications.
Known mitigations
Turn off by setting driver module parameter use_persistent_mapping=0 and reloading the driver.
Fixed gdrcopy version
2.4.4
- Dominant language
- C
- Stars
- 1.4k
- Forks
- 198
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/gdrcopy
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Proposal: honor `TMPDIR` in gdrcopy package build scriptsPossibly taken @efegokdemir claimed this 53 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 58/100
Similar issues
-
SPIDevice selects an active-high device as soon as it is createdPossibly taken @mikeysklar claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
adafruit/circuitpython#11869 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
gnif/LookingGlass#1352 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
bazel-contrib/rules_go#4784 · 1 comment ·
Maintainers usually reply within 1 day