arm64 WSL2: whole VM stalls for minutes when many threads page-fault a cold mmap'd file

Đang mở
  • Có gì không ổn?
#42,200 4 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
38/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
c
Lĩnh vực
operating-systems

Hướng nghiên cứu

The report identifies arm64 Linux spinlocks and Hyper-V early initialization but names no WSL source files or tests. Start by locating the arm64 spinlock fallback and Hyper-V initialization paths in the WSL kernel tree, then compare them with x86's virt_spin_lock() and Hyper-V setup. Done means the reported multithreaded mmap repro no longer stalls the VM, while the proposed static key remains off outside Hyper-V.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

wsl2
Windows Version

Windows 11 [Version 10.0.28120.2738]

WSL Version

3.0.3.0

Are you using WSL 1, WSL 2, or WSLC?
  • WSL 2
  • WSL 1
  • WSLC
Kernel Version

6.18.54.1-microsoft-standard-WSL2

Distro Version

Ubuntu-24.04

Other Software

No response

Repro Steps
What happens

On an arm64 WSL2 VM with many vCPUs, the whole VM can stall for minutes when many threads page-fault the same file-backed mmap at the same time and the file isn't in the page cache yet.

PyTorch apps can hit this issue. When they load model weights, they copy tensors out of an mmap'd file, and PyTorch splits each copy across all cores by default. ComfyUI is one example: on a 72-vCPU VM it hangs while loading its text encoder.

This isn't just a slow app. Once it starts, the rest of the VM stops making progress too:

  • The guest kernel logs repeated RCU stalls: 17 stall reports in about an hour. Each one lists CPUs that have been stuck for at least 60 s in the page-fault path (do_page_fault → filemap_fault → filemap_add_folio), up to 29 CPUs at a time.
  • systemd's watchdog kills systemd-resolved and systemd-udevd after they stop responding for 3 minutes.
  • Shell commands in the VM hang.

Guest journal excerpt from one of these stalls (stall reports, a representative backtrace and the watchdog kills): wsl-arm64-rcu-stall-excerpt.log

With fewer threads, the same work finishes in well under a second.

Environment
  • 72-core arm64 host
  • Windows 11 build 28120.2738 (arm64)
  • WSL 3.0.1. WSL 2.7.x has the same problem.
  • Stock kernel 6.18.54.1-microsoft-standard-WSL2 (attached log)
  • Default Windows and WSL settings (the VM gets all 72 vCPUs)
Repro

The script below needs only PyTorch, and no root. It writes a 512 MiB file, evicts it from the page cache, and copies it out of a read-only mmap in 32 MiB tensors, the same way PyTorch apps load model weights. It runs once with 4 threads and once with one thread per vCPU.

pip install torch --index-url https://download.pytorch.org/whl/cpu
python3 mmap_fault_repro.py          # 4 threads, then one per vCPU
python3 mmap_fault_repro.py 4 16 72  # or pick thread counts
mmap_fault_repro.py 
#!/usr/bin/env python3
"""Many threads page-faulting the same cold file mmap stall an arm64 WSL2 VM.

Copies a 512 MiB file out of a read-only mmap with PyTorch, the way PyTorch apps
load model weights: each 32 MiB tensor is copied with tensor.copy_(), which splits
the copy across all threads. The file is evicted from the page cache first, so the
threads page-fault it in, all within the same tensor at a time.

    pip install torch --index-url https://download.pytorch.org/whl/cpu
    python3 mmap_fault_repro.py [THREADS ...]    # default: 4, then one per CPU

72-vCPU arm64 WSL2 VM: 4 threads take ~0.1 s, 72 threads take minutes (almost all system time).
"""
import mmap
import os
import platform
import resource
import sys
import time
import warnings

import torch

PATH = os.path.expanduser("~/mmap_fault_repro.bin")  # on the Linux filesystem, not /mnt/c
TENSORS, TENSOR_BYTES = 16, 32 << 20

warnings.filterwarnings("ignore", "The given buffer is not writable")


def make_file():
    if os.path.exists(PATH) and os.path.getsize(PATH) == TENSORS * TENSOR_BYTES:
        return
    with open(PATH, "wb") as f:
        for _ in range(TENSORS):
            f.write(bytes(TENSOR_BYTES))
        os.fsync(f.fileno())  # dirty pages can't be evicted


def copy_from_cold_mmap(threads):
    torch.set_num_threads(threads)
    with open(PATH, "rb") as f:
        os.posix_fadvise(f.fileno(), 0, 0, os.POSIX_FADV_DONTNEED)  # evict from the page cache
        m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    src = [torch.frombuffer(m, dtype=torch.float16, count=TENSOR_BYTES // 2, offset=i * TENSOR_BYTES)
           for i in range(TENSORS)]
    dst = [torch.empty_like(s) for s in src]

    r0, t0 = resource.getrusage(resource.RUSAGE_SELF), time.monotonic()
    for s, d in zip(src, dst):
        d.copy_(s)
    wall, r1 = time.monotonic() - t0, resource.getrusage(resource.RUSAGE_SELF)
    print(f"threads={threads:<3}  wall={wall:7.2f} s  sys={r1.ru_stime - r0.ru_stime:8.1f} s", flush=True)


def main():
    print(f"kernel {platform.release()} {platform.machine()}, {os.cpu_count()} CPUs, torch {torch.__version__}")
    make_file()
    for threads in [int(a) for a in sys.argv[1:]] or [4, os.cpu_count()]:
        copy_from_cold_mmap(threads)


if __name__ == "__main__":
    main()

Output on the stock kernel:

kernel 6.18.40.1-microsoft-standard-WSL2 aarch64, 72 CPUs, torch 2.14.1+cu130
threads=4    wall=   0.27 s  sys=     0.4 s
threads=72   wall= 308.26 s  sys= 13775.2 s

Across runs, 4 threads take 0.1–0.3 s. 72 threads take 110–310 s, and the system time adds up to 5,000–13,800 CPU-seconds. Nearly all of that time is vCPUs spinning in the kernel.

Why it happens

All of these page faults go through the same kernel spinlock, the one that protects the file's page cache. Here is the chain of events:

  1. arm64 Linux spinlocks are strictly FIFO. When the lock is released, it goes to the next waiter in line, and only that waiter can take it.
  2. In a VM, that waiter might not be running. The host schedules vCPUs like ordinary threads, so any vCPU can be descheduled at any moment.
  3. If the next waiter isn't running, nobody gets the lock. Every other vCPU keeps spinning until the host runs that one vCPU again. A handoff that normally takes nanoseconds now takes milliseconds.
  4. With dozens of vCPUs queued up, this compounds. The VM spends almost all its time spinning, and anything else that needs the same lock, or just needs a CPU, stalls too. That's where the RCU stalls and watchdog kills come from.

x86 guests don't have this problem. Under Hyper-V, x86 Linux uses Hyper-V's paravirtualized spinlocks when the host supports them: a waiting vCPU halts and is woken by the hypervisor instead of spinning. Otherwise it falls back to virt_spin_lock(), a simple unfair test-and-set lock that x86 Linux uses under any hypervisor. Any running vCPU can grab that lock. Either way, a descheduled waiter doesn't block everyone else.

arm64 has neither. Hyper-V has no paravirtualized spinlocks for arm64 guests, and arm64 Linux has no virt_spin_lock() fallback. So arm64 WSL2 is stuck with strict FIFO handoff and no protection against preempted waiters.

Proposed fix

Give arm64 the same fallback x86 has, and turn it on under Hyper-V:

  • Add an arm64 virt_spin_lock() that mirrors x86's test-and-set fallback, behind a static key that is off by default. Bare-metal arm64 and other hypervisors see no change.
  • Turn the key on in the arm64 Hyper-V early init, before secondary CPUs come up.

Longer term, it may be worth looking at Hyper-V paravirtualized spinlocks for arm64 guests, like hv_init_spinlocks() on x86. Waiting vCPUs would then halt instead of spinning, and arm64 guests would behave the same as x86 guests under Hyper-V.

Workarounds until then

These make the stall less likely, but they don't prevent it:

  • Limit the VM's vCPUs with processors= in .wslconfig.
  • Limit thread counts in the app (OMP_NUM_THREADS, torch.set_num_threads()).
Expected Behavior

The VM should not hang when many threads page-fault a cold mmap'd file at the same time.

Actual Behavior

The whole VM hangs for minutes, with RCU stall warnings and systemd watchdog kills (see the description above).

Diagnostic Logs

Journal log during the time that the repro script ran:

wsl-arm64-rcu-stall-excerpt.log

Ngôn ngữ chính
C++
Star
34.3k
Fork
1.9k
Merge trung bình
1 ngày 13 giờ
Pull request đã merge (30 ngày)
166

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của microsoft/WSL

Tất cả issue của microsoft/WSL

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.