Why do CUDA device allocations appear to produce a nearly 1:1 increase in WSL host committed memory? Is this expected behavior related to CUDA/WSL2 GPU memory management, or an actual memory-accounting issue?

Abierto
#42,294 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python, pytorch

Línea de trabajo

Start with the minimal PyTorch reproduction in the issue and compare its CUDA allocation steps with the reported Windows vmmemWSL commit measurements. The report points to CUDA allocation backing or reservation in WSL2 GPU paravirtualization, but names no source files or tests. First determine whether this commit growth is expected; done means documenting the behavior or identifying a reproducible accounting defect.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

GPU wsl2
Windows Version

Microsoft Windows [Version 10.0.26200.7840]

WSL Version

3.0.1.0

Are you using WSL 1, WSL 2, or WSLC?
  • WSL 2
  • WSL 1
  • WSLC
Kernel Version

6.18.40.1-microsoft-standard-WSL2

Distro Version

Ubuntu 24.04.3 LTS (Noble Numbat)

Other Software

Hardware:

  • Primary GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB VRAM, sm_120)
  • Secondary GPU: NVIDIA GeForce RTX 4090 (24 GB VRAM)
  • NVIDIA Windows Driver: 617.42
  • System RAM: 128 GiB

Software:

  • Windows 11, Build 26200.7840
  • WSL: 3.0.1.0
  • WSL2 Kernel: 6.18.40.1-microsoft-standard-WSL2
  • Ubuntu 24.04.3 LTS (Noble Numbat)
  • PyTorch: 2.9.0+cu128 (verified in native Windows environment; WSL environment version not yet confirmed)
  • CUDA runtime (Windows PyTorch): 12.8

WSL2 configuration (%UserProfile%\.wslconfig):

[wsl2]
networkingMode=nat
guiApplications=false
memory=64GB
swap=32GB

[experimental]
autoMemoryReclaim=gradual

The issue occurs during CUDA GPU allocations under WSL2 and can be reproduced using a minimal PyTorch script without a full model or training framework.

Repro Steps
Steps to reproduce

1. Configure WSL2

Configure .wslconfig as described above, then restart WSL using:

wsl --shutdown

Launch Ubuntu and activate a Linux Python environment with CUDA-enabled PyTorch installed.

2. Record initial WSL memory information

Run from Windows PowerShell:

wsl -d Ubuntu -- sh -c 'grep -E "^(MemTotal|MemFree|MemAvailable|Cached|SwapTotal|SwapFree|Committed_AS):" /proc/meminfo'

Observed WSL memory statistics:

MemTotal:       65845096 kB
MemFree:        37435408 kB
MemAvailable:   50050264 kB
Cached:         11298936 kB
SwapTotal:      33554432 kB
SwapFree:       33554432 kB
Committed_AS:   19508596 kB

3. Run a Windows-side commit monitor

Open a separate Windows PowerShell terminal:

while ($true) {
    $vm = Get-Process vmmemWSL -ErrorAction SilentlyContinue
    $sys = (Get-Counter '\Memory\Committed Bytes').CounterSamples[0].CookedValue

    "{0:HH:mm:ss} System={1:N2}GiB WSLCommit={2:N2}GiB WSLWorkingSet={3:N2}GiB" -f `
        (Get-Date),
        ($sys / 1GB),
        ($vm.PrivateMemorySize64 / 1GB),
        ($vm.WorkingSet64 / 1GB)

    Start-Sleep -Seconds 3
}

This monitors:

  • Total Windows system committed memory
  • vmmemWSL private committed memory
  • vmmemWSL working set

4. Execute the minimal CUDA allocation test inside Ubuntu WSL2

Create repro_cuda_commit.py:

import gc
import time
import torch

print("GPU:", torch.cuda.get_device_name(0), flush=True)

chunks = []

def measure(stage):
    torch.cuda.synchronize()

    print(
        f"{stage}: GPU allocated="
        f"{torch.cuda.memory_allocated()/1024**3:.2f} GiB",
        flush=True
    )

    with open("/proc/meminfo") as f:
        for line in f:
            if line.startswith(("MemAvailable:", "Committed_AS:")):
                print(line.strip(), flush=True)

    time.sleep(15)

measure("CUDA initialized")

for i in range(3):
    x = torch.empty(
        4 * 1024**3,
        device="cuda:0",
        dtype=torch.uint8
    )
    x.fill_(1)
    chunks.append(x)

    measure(f"{(i+1)*4} GiB GPU allocation")

chunks.clear()
del x
gc.collect()
torch.cuda.empty_cache()

measure("GPU allocations released")

Run:

python repro_cuda_commit.py

5. Observe Windows host commit

While the script executes, observe the WSLCommit column in the Windows PowerShell monitor.

GPU memory is allocated in three consecutive 4 GiB increments (4, 8, and 12 GiB total), followed by complete release.

Compare the increase in GPU VRAM allocation against the corresponding increase in vmmemWSL private committed memory.

6. Check Windows page file usage

Get-CimInstance Win32_PageFileUsage |
    Format-Table Name, AllocatedBaseSize, CurrentUsage, PeakUsage

This helps distinguish Windows commit accounting from actual page file occupancy.

Expected Behavior

CUDA device-only allocations (torch.empty(..., device="cuda:0")) should primarily consume GPU device memory, without requiring an approximately equal increase in Windows host committed memory unless this is an intentional, documented requirement of WSL2 GPU virtualization.

Specifically:

  1. Allocating 4 GiB of GPU VRAM should not unexpectedly require approximately 4 GiB of additional Windows system commit.
  2. Large GPU workloads should not encounter host commit-limit pressure solely because GPU VRAM allocations are being mirrored into Windows commit accounting, unless such accounting is required for the virtualization architecture.
  3. The relationship between device VRAM allocations, vmmemWSL private committed memory, and Windows commit limits should be documented if this behavior is expected.
  4. Users should not need to continuously increase the Windows page file size simply to accommodate GPU allocations without a corresponding explanation of the host commit requirement.

The goal is to understand whether this near-1:1 host commit growth is expected WSL2/CUDA virtualization behavior or an unnecessary memory-accounting overhead that can be corrected.

Actual Behavior
CUDA device allocations produce approximately 1:1 Windows host commit growth

Running the minimal reproduction on the NVIDIA RTX PRO 6000 Blackwell under WSL2 produces the following measurements:

GPU VRAM allocated | vmmemWSL private commit | Increase from baseline -- | -- | -- 0 GiB | 8.99 GiB | — 4 GiB | 13.12 GiB | +4.13 GiB 8 GiB | 17.13 GiB | +8.14 GiB 12 GiB | 21.10 GiB | +12.11 GiB Released | 9.28 GiB | +0.29 GiB
Findings

1. Nearly exact 1:1 relationship

Every additional 4 GiB of CUDA device allocation produces approximately 4 GiB of additional vmmemWSL private committed memory.

At 12 GiB of GPU allocations, WSL host commit has increased by 12.11 GiB.

This strongly suggests the GPU allocation size is directly associated with the additional Windows host commit charge.

2. Memory commitment is reversible

After executing:

chunks.clear()
del x
gc.collect()
torch.cuda.empty_cache()

WSL committed memory falls from 21.10 GiB to 9.28 GiB, close to the initial 8.99 GiB baseline.

This makes the behavior reproducible and allocation-dependent rather than evidence of a monotonically growing memory leak.

3. Windows page file occupancy remains low

Separately observed Windows page file statistics:

Name             AllocatedBaseSize CurrentUsage PeakUsage
----             ----------------- ------------ ---------
C:\pagefile.sys              32768          468       850

All sizes are in MiB.

The page file has 32 GiB allocated, but only approximately 468 MiB was in use at the time of measurement. This illustrates that the observed increase in committed memory is not equivalent to an equal increase in actual page file occupancy.

4. Impact during model training

During larger PyTorch model training workloads on the RTX PRO 6000 Blackwell, total Windows system committed memory has exceeded 70 GB.

The controlled test demonstrates that CUDA VRAM allocations contribute directly to vmmemWSL host commit.

This reduces available Windows commit headroom and can make the system dependent on larger page file configurations even when the additional memory is not actively being paged to disk.

Suspected area

The measurements suggest an interaction between CUDA GPU device allocations, WSL2 GPU virtualization, and Windows host commit accounting.

Possible areas for investigation include CUDA allocation backing or reservation behavior in the WSL2 GPU paravirtualization path.

The test does not establish whether physical host RAM is duplicated or whether the additional charge represents reserved virtual memory.

Question: Is the approximately 1:1 host commit charge for CUDA device allocations an intentional part of WSL2's GPU memory model, or can it be eliminated or reduced?

Diagnostic Logs
System information
Windows Version: 10.0.26200.7840
WSL Version: 3.0.1.0
WSL2 Kernel: 6.18.40.1-microsoft-standard-WSL2
Distribution: Ubuntu 24.04.3 LTS

GPU 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
GPU 1: NVIDIA GeForce RTX 4090
NVIDIA Driver: 617.42

Windows PyTorch: 2.9.0+cu128
Windows PyTorch CUDA: 12.8
Windows page file information
Name             AllocatedBaseSize CurrentUsage PeakUsage
----             ----------------- ------------ ---------
C:\pagefile.sys              32768          468       850
Controlled GPU allocation results
GPU allocated    WSL private commit    Delta
0 GiB             8.99 GiB            Baseline
4 GiB            13.12 GiB            +4.13 GiB
8 GiB            17.13 GiB            +8.14 GiB
12 GiB           21.10 GiB            +12.11 GiB
Released          9.28 GiB            +0.29 GiB
Related issue

This report isolates a different observable behavior from the related CUDA context overhead issue: proportional Windows host commit growth during explicit CUDA device allocations, independently of model loading or inference frameworks.

The minimal PyTorch reproduction is included in the Repro Steps section.

Additional WSL-side PyTorch version information and WSL diagnostic logs can be provided if requested.

Lenguaje dominante
C++
Estrellas
34.4k
Forks
1.9k
Merge medio
1 d 23 h
PR fusionados (30 d)
166

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/WSL

Todos los issues de microsoft/WSL

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.