ARM64: threads stuck forever on a store instruction with hv_balloon free page reporting (rewriting the same byte via /proc/PID/mem unsticks them)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- linux
- Domain
- backend, operating-systems
Research direction
Start by examining the ARM64 WSL2 guest kernel paths implicated in the report: hv_balloon free page reporting, user page faults, and the kernel-side do_swap_page trace. Compare behavior with balloon reporting disabled and collect the offered per-incident traces, especially host-side Hyper-V ETW during a stuck store. Done means identifying and fixing the cause so guest stores complete or fault and recover under memory pressure.
Written by the indexing model from the issue text.
Description
Windows Version
10.0.26300.9818 (Snapdragon X2 Elite, ARM64, 48 GB RAM)
WSL Version
3.0.5.0 (also seen on 2.7.11 before upgrading)
Are you using WSL 1 or WSL 2?
- WSL 2
Kernel Version
6.18.58.1-microsoft-standard-WSL2 (aarch64, PREEMPT_DYNAMIC)
Distro Version
Ubuntu 26.04 LTS
Other Software
Docker Desktop (engine 29.9.2, WSL integration), Node.js tools (Claude Code, MCP servers), PostgreSQL/MariaDB/PHP containers.
.wslconfig:
[wsl2]
memory=20GB
processors=12
swap=8GB
kernelCommandLine=hugepages=1300 preempt=full
[experimental]
autoMemoryReclaim=gradual
sparseVhd=true
Repro Steps
No deterministic repro. It happens several times a day on a busy dev VM (Docker containers, many Node processes, memory pressure that makes the balloon / free page reporting active):
- ARM64 Windows host, WSL 2 with
hv_balloonloaded (default) and free page reporting active (autoMemoryReclaim=gradual, or the defaultpage_reporting_order). - Run a normal multi-process workload for a few hours (Docker containers, Node, Postgres).
- Sooner or later a thread sits at 100% CPU forever on a single store instruction.
Expected Behavior
A user-space store to a mapped, writable page completes, or faults and gets resolved by the guest kernel.
Actual Behavior
A user-space thread gets stuck forever on one store instruction.
-
The thread spins at 100% user time with zero voluntary context switches, and its PC never changes. We sampled it with gdb: two reads 2 s apart, over minutes to hours.
-
The instruction is always a write to memory:
When (UTC-3) Process Instruction at stuck PC 2026-10-31 postgres backends, soketi str/stp2026-11-04 14:03 node (JIT worker) str q3, [x0]insidememcpy, 10 min2026-11-04 (afternoon) node (hooks thread) dc zvainsidememset, ~2 h2026-11-04 (afternoon) codex (Rust, tokio worker) ldadd2026-11-04 16:54 node str q3, [x0]2026-11-04 18:39 node (MCP server) str x3, [x2]2026-11-04 18:41 same process, V8 worker cas x0, x1, [x2]2026-11-04 18:42 another node stur w2, [x1, #19] -
The page is never resolved for that thread. If we write the same byte that is already at the target address through
/proc/PID/mem, the kernel resolves the page from outside (access_remote_vm). The stuck thread then continues in under 3 s: state goes fromRtoSand voluntary switches start to count again. We now do this automatically and it has worked in every case above:python3 -I -c "f=open('/proc/$PID/mem','r+b',buffering=0); f.seek($ADDR); b=f.read(1); f.seek($ADDR); f.write(b)"To us this points to a stale or lost stage-2 mapping or a permission state in the hypervisor for a guest page. The guest believes the page is present and writable, so it never faults it in again. A host-side write path re-establishes it. In our history the problem correlates with
hv_balloonfree page reporting (pages reported to the host and later reused by the guest):- With
initcall_blacklist=init_balloon_drv(balloon disabled, 2026-11-01 18:29 to 2026-11-03 10:51, ~40 h) we saw no stuck store. The only hot threads then were ordinary user-space loops with a changing PC. With the balloon back on, the first kernel-side case came 11.5 h later. - With
page_reporting.page_reporting_order=10it still happened. - Upgrading from WSL 2.7.11 to 3.0.5 did not change it.
- With
Kernel-side variant. The same thing can happen to a store done by the kernel on behalf of a task. On 2026-11-03 22:23:
- A task's
stimegrew by ~100 ticks per second and itsutimestayed constant. - The task was a Node V8 worker thread on CPU 4. The
sysrq-lNMI backtrace showeddo_swap_page→folio_add_new_anon_rmap→mod_memcg_lruvec_state+0x18c(insidememcg_rstat_updated, preemption disabled, a per-cpuldadd). - With the default
PREEMPT(none), that CPU never released the migration stopper. Every latersched_setaffinity/ migration (affine_move_task) piled up behind it. - Load reached 458, processes became unkillable (SIGKILL ignored), and only
wsl --shutdownrecovered the VM. - Booting with
preempt=fullcontains it: one CPU is lost instead of the whole VM. The root cause is still there.
Diagnostic Logs
We keep per-incident dossiers (gdb disassembly around the PC, registers, /proc/PID/task/*/stat|status, sysrq-l/sysrq-w traces, /proc/interrupts, free, cmdline) and can attach them or run collect-wsl-logs.ps1 during the next occurrence if that helps. Please tell us which traces would be most useful, for example Hyper-V ETW providers on the host while a thread is stuck.
Workaround for others hitting this on ARM64: either disable the balloon (kernelCommandLine=initcall_blacklist=init_balloon_drv, at the cost of memory never returning to Windows), or detect threads stuck on a store and rewrite the same byte through /proc/PID/mem as above, plus preempt=full so a kernel-side occurrence doesn't freeze the whole VM.
- Dominant language
- C++
- Stars
- 34.4k
- Forks
- 1.9k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 166
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/WSL
-
feature wsl2
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
microsoft/WSL#42306 · 3 comments ·
Maintainers usually reply within 1 day
-
feature
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
microsoft/WSL#42114 · 3 comments ·
Maintainers usually reply within 1 day
-
feature kernel wsl2
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
microsoft/WSL#41107 · 3 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
feature kernel
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
microsoft/WSL#41054 · 9 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
feature GPU
Difficulty 1/5 1-3 hours Newbie friendliness 70/100
microsoft/WSL#13592 · 4 comments ·
Maintainers usually reply within 1 day
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
nextcloud/passman-android#214 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
2026.11.0 fails to install on Intel macOS: requirements.txt pins cryptography==50.0.2 for Darwin x86_64May be free again A pull request for this issue was closed without being merged. Open
Difficulty 1/5 Under an hour Newbie friendliness 87/100
esphome/esphome#22892 · 1 comment ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
distante/esphome-components#50 · 1 comment ·
Maintainers usually reply within 1 day
-
arch-based hyprland
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
noctalia-dev/noctalia#5423 ·
Maintainers usually reply within 1 day