ARM64: threads stuck forever on a store instruction with hv_balloon free page reporting (rewriting the same byte via /proc/PID/mem unsticks them)

Open
#42,174 9 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
linux

Research direction

Start by examining the ARM64 WSL2 guest kernel paths implicated in the report: hv_balloon free page reporting, user page faults, and the kernel-side do_swap_page trace. Compare behavior with balloon reporting disabled and collect the offered per-incident traces, especially host-side Hyper-V ETW during a stuck store. Done means identifying and fixing the cause so guest stores complete or fault and recover under memory pressure.

Written by the indexing model from the issue text.

Description

emailed-logs hypervisor-platform wsl2
Windows Version

10.0.26300.9818 (Snapdragon X2 Elite, ARM64, 48 GB RAM)

WSL Version

3.0.5.0 (also seen on 2.7.11 before upgrading)

Are you using WSL 1 or WSL 2?
  • WSL 2
Kernel Version

6.18.58.1-microsoft-standard-WSL2 (aarch64, PREEMPT_DYNAMIC)

Distro Version

Ubuntu 26.04 LTS

Other Software

Docker Desktop (engine 29.9.2, WSL integration), Node.js tools (Claude Code, MCP servers), PostgreSQL/MariaDB/PHP containers.

.wslconfig:

[wsl2]
memory=20GB
processors=12
swap=8GB
kernelCommandLine=hugepages=1300 preempt=full

[experimental]
autoMemoryReclaim=gradual
sparseVhd=true
Repro Steps

No deterministic repro. It happens several times a day on a busy dev VM (Docker containers, many Node processes, memory pressure that makes the balloon / free page reporting active):

  1. ARM64 Windows host, WSL 2 with hv_balloon loaded (default) and free page reporting active (autoMemoryReclaim=gradual, or the default page_reporting_order).
  2. Run a normal multi-process workload for a few hours (Docker containers, Node, Postgres).
  3. Sooner or later a thread sits at 100% CPU forever on a single store instruction.
Expected Behavior

A user-space store to a mapped, writable page completes, or faults and gets resolved by the guest kernel.

Actual Behavior

A user-space thread gets stuck forever on one store instruction.

  • The thread spins at 100% user time with zero voluntary context switches, and its PC never changes. We sampled it with gdb: two reads 2 s apart, over minutes to hours.

  • The instruction is always a write to memory:

    When (UTC-3) Process Instruction at stuck PC
    2026-10-31 postgres backends, soketi str / stp
    2026-11-04 14:03 node (JIT worker) str q3, [x0] inside memcpy, 10 min
    2026-11-04 (afternoon) node (hooks thread) dc zva inside memset, ~2 h
    2026-11-04 (afternoon) codex (Rust, tokio worker) ldadd
    2026-11-04 16:54 node str q3, [x0]
    2026-11-04 18:39 node (MCP server) str x3, [x2]
    2026-11-04 18:41 same process, V8 worker cas x0, x1, [x2]
    2026-11-04 18:42 another node stur w2, [x1, #19]
  • The page is never resolved for that thread. If we write the same byte that is already at the target address through /proc/PID/mem, the kernel resolves the page from outside (access_remote_vm). The stuck thread then continues in under 3 s: state goes from R to S and voluntary switches start to count again. We now do this automatically and it has worked in every case above:

    python3 -I -c "f=open('/proc/$PID/mem','r+b',buffering=0); f.seek($ADDR); b=f.read(1); f.seek($ADDR); f.write(b)"
    

    To us this points to a stale or lost stage-2 mapping or a permission state in the hypervisor for a guest page. The guest believes the page is present and writable, so it never faults it in again. A host-side write path re-establishes it. In our history the problem correlates with hv_balloon free page reporting (pages reported to the host and later reused by the guest):

    • With initcall_blacklist=init_balloon_drv (balloon disabled, 2026-11-01 18:29 to 2026-11-03 10:51, ~40 h) we saw no stuck store. The only hot threads then were ordinary user-space loops with a changing PC. With the balloon back on, the first kernel-side case came 11.5 h later.
    • With page_reporting.page_reporting_order=10 it still happened.
    • Upgrading from WSL 2.7.11 to 3.0.5 did not change it.

Kernel-side variant. The same thing can happen to a store done by the kernel on behalf of a task. On 2026-11-03 22:23:

  • A task's stime grew by ~100 ticks per second and its utime stayed constant.
  • The task was a Node V8 worker thread on CPU 4. The sysrq-l NMI backtrace showed do_swap_page → folio_add_new_anon_rmap → mod_memcg_lruvec_state+0x18c (inside memcg_rstat_updated, preemption disabled, a per-cpu ldadd).
  • With the default PREEMPT(none), that CPU never released the migration stopper. Every later sched_setaffinity / migration (affine_move_task) piled up behind it.
  • Load reached 458, processes became unkillable (SIGKILL ignored), and only wsl --shutdown recovered the VM.
  • Booting with preempt=full contains it: one CPU is lost instead of the whole VM. The root cause is still there.
Diagnostic Logs

We keep per-incident dossiers (gdb disassembly around the PC, registers, /proc/PID/task/*/stat|status, sysrq-l/sysrq-w traces, /proc/interrupts, free, cmdline) and can attach them or run collect-wsl-logs.ps1 during the next occurrence if that helps. Please tell us which traces would be most useful, for example Hyper-V ETW providers on the host while a thread is stuck.

Workaround for others hitting this on ARM64: either disable the balloon (kernelCommandLine=initcall_blacklist=init_balloon_drv, at the cost of memory never returning to Windows), or detect threads stuck on a store and rewrite the same byte through /proc/PID/mem as above, plus preempt=full so a kernel-side occurrence doesn't freeze the whole VM.

Dominant language
C++
Stars
34.4k
Forks
1.9k
Avg merge
1d 23h
Merged PRs (30d)
166

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/WSL

All issues in microsoft/WSL

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.