ARM64: threads stuck forever on a store instruction with hv_balloon free page reporting (rewriting the same byte via /proc/PID/mem unsticks them)

Aperta
#42,174 9 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
linux

Direzione di ricerca

Start by examining the ARM64 WSL2 guest kernel paths implicated in the report: hv_balloon free page reporting, user page faults, and the kernel-side do_swap_page trace. Compare behavior with balloon reporting disabled and collect the offered per-incident traces, especially host-side Hyper-V ETW during a stuck store. Done means identifying and fixing the cause so guest stores complete or fault and recover under memory pressure.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

emailed-logs hypervisor-platform wsl2
Windows Version

10.0.26300.9818 (Snapdragon X2 Elite, ARM64, 48 GB RAM)

WSL Version

3.0.5.0 (also seen on 2.7.11 before upgrading)

Are you using WSL 1 or WSL 2?
  • WSL 2
Kernel Version

6.18.58.1-microsoft-standard-WSL2 (aarch64, PREEMPT_DYNAMIC)

Distro Version

Ubuntu 26.04 LTS

Other Software

Docker Desktop (engine 29.9.2, WSL integration), Node.js tools (Claude Code, MCP servers), PostgreSQL/MariaDB/PHP containers.

.wslconfig:

[wsl2]
memory=20GB
processors=12
swap=8GB
kernelCommandLine=hugepages=1300 preempt=full

[experimental]
autoMemoryReclaim=gradual
sparseVhd=true
Repro Steps

No deterministic repro. It happens several times a day on a busy dev VM (Docker containers, many Node processes, memory pressure that makes the balloon / free page reporting active):

  1. ARM64 Windows host, WSL 2 with hv_balloon loaded (default) and free page reporting active (autoMemoryReclaim=gradual, or the default page_reporting_order).
  2. Run a normal multi-process workload for a few hours (Docker containers, Node, Postgres).
  3. Sooner or later a thread sits at 100% CPU forever on a single store instruction.
Expected Behavior

A user-space store to a mapped, writable page completes, or faults and gets resolved by the guest kernel.

Actual Behavior

A user-space thread gets stuck forever on one store instruction.

  • The thread spins at 100% user time with zero voluntary context switches, and its PC never changes. We sampled it with gdb: two reads 2 s apart, over minutes to hours.

  • The instruction is always a write to memory:

    When (UTC-3) Process Instruction at stuck PC
    2026-10-31 postgres backends, soketi str / stp
    2026-11-04 14:03 node (JIT worker) str q3, [x0] inside memcpy, 10 min
    2026-11-04 (afternoon) node (hooks thread) dc zva inside memset, ~2 h
    2026-11-04 (afternoon) codex (Rust, tokio worker) ldadd
    2026-11-04 16:54 node str q3, [x0]
    2026-11-04 18:39 node (MCP server) str x3, [x2]
    2026-11-04 18:41 same process, V8 worker cas x0, x1, [x2]
    2026-11-04 18:42 another node stur w2, [x1, #19]
  • The page is never resolved for that thread. If we write the same byte that is already at the target address through /proc/PID/mem, the kernel resolves the page from outside (access_remote_vm). The stuck thread then continues in under 3 s: state goes from R to S and voluntary switches start to count again. We now do this automatically and it has worked in every case above:

    python3 -I -c "f=open('/proc/$PID/mem','r+b',buffering=0); f.seek($ADDR); b=f.read(1); f.seek($ADDR); f.write(b)"
    

    To us this points to a stale or lost stage-2 mapping or a permission state in the hypervisor for a guest page. The guest believes the page is present and writable, so it never faults it in again. A host-side write path re-establishes it. In our history the problem correlates with hv_balloon free page reporting (pages reported to the host and later reused by the guest):

    • With initcall_blacklist=init_balloon_drv (balloon disabled, 2026-11-01 18:29 to 2026-11-03 10:51, ~40 h) we saw no stuck store. The only hot threads then were ordinary user-space loops with a changing PC. With the balloon back on, the first kernel-side case came 11.5 h later.
    • With page_reporting.page_reporting_order=10 it still happened.
    • Upgrading from WSL 2.7.11 to 3.0.5 did not change it.

Kernel-side variant. The same thing can happen to a store done by the kernel on behalf of a task. On 2026-11-03 22:23:

  • A task's stime grew by ~100 ticks per second and its utime stayed constant.
  • The task was a Node V8 worker thread on CPU 4. The sysrq-l NMI backtrace showed do_swap_page → folio_add_new_anon_rmap → mod_memcg_lruvec_state+0x18c (inside memcg_rstat_updated, preemption disabled, a per-cpu ldadd).
  • With the default PREEMPT(none), that CPU never released the migration stopper. Every later sched_setaffinity / migration (affine_move_task) piled up behind it.
  • Load reached 458, processes became unkillable (SIGKILL ignored), and only wsl --shutdown recovered the VM.
  • Booting with preempt=full contains it: one CPU is lost instead of the whole VM. The root cause is still there.
Diagnostic Logs

We keep per-incident dossiers (gdb disassembly around the PC, registers, /proc/PID/task/*/stat|status, sysrq-l/sysrq-w traces, /proc/interrupts, free, cmdline) and can attach them or run collect-wsl-logs.ps1 during the next occurrence if that helps. Please tell us which traces would be most useful, for example Hyper-V ETW providers on the host while a thread is stuck.

Workaround for others hitting this on ARM64: either disable the balloon (kernelCommandLine=initcall_blacklist=init_balloon_drv, at the cost of memory never returning to Windows), or detect threads stuck on a store and rewrite the same byte through /proc/PID/mem as above, plus preempt=full so a kernel-side occurrence doesn't freeze the whole VM.

Lingua principale
C++
Stelle
34.4k
Fork
1.9k
Merge medio
1g 23h
PR unite (30g)
166

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/WSL

Tutte le issue di microsoft/WSL

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.