ARM64: threads stuck forever on a store instruction with hv_balloon free page reporting (rewriting the same byte via /proc/PID/mem unsticks them)
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 25/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- linux
- Ambito
- backend, operating-systems
Direzione di ricerca
Start by examining the ARM64 WSL2 guest kernel paths implicated in the report: hv_balloon free page reporting, user page faults, and the kernel-side do_swap_page trace. Compare behavior with balloon reporting disabled and collect the offered per-incident traces, especially host-side Hyper-V ETW during a stuck store. Done means identifying and fixing the cause so guest stores complete or fault and recover under memory pressure.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Windows Version
10.0.26300.9818 (Snapdragon X2 Elite, ARM64, 48 GB RAM)
WSL Version
3.0.5.0 (also seen on 2.7.11 before upgrading)
Are you using WSL 1 or WSL 2?
- WSL 2
Kernel Version
6.18.58.1-microsoft-standard-WSL2 (aarch64, PREEMPT_DYNAMIC)
Distro Version
Ubuntu 26.04 LTS
Other Software
Docker Desktop (engine 29.9.2, WSL integration), Node.js tools (Claude Code, MCP servers), PostgreSQL/MariaDB/PHP containers.
.wslconfig:
[wsl2]
memory=20GB
processors=12
swap=8GB
kernelCommandLine=hugepages=1300 preempt=full
[experimental]
autoMemoryReclaim=gradual
sparseVhd=true
Repro Steps
No deterministic repro. It happens several times a day on a busy dev VM (Docker containers, many Node processes, memory pressure that makes the balloon / free page reporting active):
- ARM64 Windows host, WSL 2 with
hv_balloonloaded (default) and free page reporting active (autoMemoryReclaim=gradual, or the defaultpage_reporting_order). - Run a normal multi-process workload for a few hours (Docker containers, Node, Postgres).
- Sooner or later a thread sits at 100% CPU forever on a single store instruction.
Expected Behavior
A user-space store to a mapped, writable page completes, or faults and gets resolved by the guest kernel.
Actual Behavior
A user-space thread gets stuck forever on one store instruction.
-
The thread spins at 100% user time with zero voluntary context switches, and its PC never changes. We sampled it with gdb: two reads 2 s apart, over minutes to hours.
-
The instruction is always a write to memory:
When (UTC-3) Process Instruction at stuck PC 2026-10-31 postgres backends, soketi str/stp2026-11-04 14:03 node (JIT worker) str q3, [x0]insidememcpy, 10 min2026-11-04 (afternoon) node (hooks thread) dc zvainsidememset, ~2 h2026-11-04 (afternoon) codex (Rust, tokio worker) ldadd2026-11-04 16:54 node str q3, [x0]2026-11-04 18:39 node (MCP server) str x3, [x2]2026-11-04 18:41 same process, V8 worker cas x0, x1, [x2]2026-11-04 18:42 another node stur w2, [x1, #19] -
The page is never resolved for that thread. If we write the same byte that is already at the target address through
/proc/PID/mem, the kernel resolves the page from outside (access_remote_vm). The stuck thread then continues in under 3 s: state goes fromRtoSand voluntary switches start to count again. We now do this automatically and it has worked in every case above:python3 -I -c "f=open('/proc/$PID/mem','r+b',buffering=0); f.seek($ADDR); b=f.read(1); f.seek($ADDR); f.write(b)"To us this points to a stale or lost stage-2 mapping or a permission state in the hypervisor for a guest page. The guest believes the page is present and writable, so it never faults it in again. A host-side write path re-establishes it. In our history the problem correlates with
hv_balloonfree page reporting (pages reported to the host and later reused by the guest):- With
initcall_blacklist=init_balloon_drv(balloon disabled, 2026-11-01 18:29 to 2026-11-03 10:51, ~40 h) we saw no stuck store. The only hot threads then were ordinary user-space loops with a changing PC. With the balloon back on, the first kernel-side case came 11.5 h later. - With
page_reporting.page_reporting_order=10it still happened. - Upgrading from WSL 2.7.11 to 3.0.5 did not change it.
- With
Kernel-side variant. The same thing can happen to a store done by the kernel on behalf of a task. On 2026-11-03 22:23:
- A task's
stimegrew by ~100 ticks per second and itsutimestayed constant. - The task was a Node V8 worker thread on CPU 4. The
sysrq-lNMI backtrace showeddo_swap_page→folio_add_new_anon_rmap→mod_memcg_lruvec_state+0x18c(insidememcg_rstat_updated, preemption disabled, a per-cpuldadd). - With the default
PREEMPT(none), that CPU never released the migration stopper. Every latersched_setaffinity/ migration (affine_move_task) piled up behind it. - Load reached 458, processes became unkillable (SIGKILL ignored), and only
wsl --shutdownrecovered the VM. - Booting with
preempt=fullcontains it: one CPU is lost instead of the whole VM. The root cause is still there.
Diagnostic Logs
We keep per-incident dossiers (gdb disassembly around the PC, registers, /proc/PID/task/*/stat|status, sysrq-l/sysrq-w traces, /proc/interrupts, free, cmdline) and can attach them or run collect-wsl-logs.ps1 during the next occurrence if that helps. Please tell us which traces would be most useful, for example Hyper-V ETW providers on the host while a thread is stuck.
Workaround for others hitting this on ARM64: either disable the balloon (kernelCommandLine=initcall_blacklist=init_balloon_drv, at the cost of memory never returning to Windows), or detect threads stuck on a store and rewrite the same byte through /proc/PID/mem as above, plus preempt=full so a kernel-side occurrence doesn't freeze the whole VM.
- Lingua principale
- C++
- Stelle
- 34.4k
- Fork
- 1.9k
- Merge medio
- 1g 23h
- PR unite (30g)
- 166
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di microsoft/WSL
-
feature wsl2
Difficoltà 2/5 1-3 ore Idoneità per principianti 80/100
microsoft/WSL#42306 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
feature
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
microsoft/WSL#42114 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
feature kernel wsl2
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
microsoft/WSL#41107 · 3 commenti · 1 reazione ·
I maintainer di solito rispondono entro 1 giorno
-
feature kernel
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
microsoft/WSL#41054 · 9 commenti · 1 reazione ·
I maintainer di solito rispondono entro 1 giorno
-
feature GPU
Difficoltà 1/5 1-3 ore Idoneità per principianti 70/100
microsoft/WSL#13592 · 4 commenti ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di microsoft/WSL
Issue simili
-
needs-format
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
Niko1221/Strata#6251 · 1 commento ·
I maintainer di solito rispondono entro 5 giorni
-
[Bug]: NP 管理器文件编辑器里删除键无效Apertabug
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
metasequoiaime/msime#11666 ·
I maintainer di solito rispondono entro 1 giorno
-
DAE2 crashes on call.without.effects with a non ref.func expressionForse già presa @stevenfontanella l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
WebAssembly/binaryen#9404 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 65/100
CodeZombie/goodluckOS#178 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
CVO-Org/immersive-cigs-rewrite#313 ·
I maintainer di solito rispondono entro 1 giorno