Observing GPU memory and/or CPU OS memory leaks with `use_persistent_mapping` enabled in `gdrdrv` during multi-process termination

Aperta
  • Qualcosa non va?
#313 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

@realarnavgoel ci sta già lavorando.

Dal 10/1/2025.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Impacted platform
All server side products, first observed on Grace-Hopper based system

Impacted gdrcopy versions
2.4.1, 2.4.2, 2.4.3

Impacted gdrcopy configs
gdrdrv driver loaded with module parameter set use_persistent_mapping=1

Scenarios
If gdrcopy persistent mapping mode is enabled,

  1. If one process opens a connection to the driver (via gdr_open), and intents to expliclity share connection (using UNIX Domain socket) with one or more processes to use the underlying connection, then the cleanup of the driver resources (via gdr_close) may be executed by one of the non-owning processes, which would be silently ignored therefore leading to CPU and GPU memory leaks.

  2. If a parent process A forks one or more child process B (instead of linux fork + exec), then connections opened by A can be attempted to be closed by B during an ungraceful termination of processes via signals (SIGSEV or SIGKILL), resulting in OS and GPU memory leaks.

By default, if persistent mode is disabled, under both scenarios, GPU resources cleanup is performed through an independent workflow in CUDA driver and hence dropping the request to close this connection is benign.

Irrespective of persistent mode, this bug may lead to small CPU kernel memory leaks.

Signature of the defect

  • On coherent platforms, e.g. Grace Hopper systems, GPU memory leaks can lead to unexpected side effects. For example, turning off the nvidia-persistenced service may hang, requiring rebooting the machine.
  • On non-coherent platforms, GPU memory leaks may reduce the functionality or performance of CUDA applications.

Known mitigations
Turn off by setting driver module parameter use_persistent_mapping=0 and reloading the driver.

Fixed gdrcopy version
2.4.4

Lingua principale
C
Stelle
1.4k
Fork
198
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/gdrcopy

Tutte le issue di NVIDIA/gdrcopy

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.