ARM64: threads stuck forever on a store instruction with hv_balloon free page reporting (rewriting the same byte via /proc/PID/mem unsticks them)

オープン
#42,174 コメント 9 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
linux

調査の方向性

Start by examining the ARM64 WSL2 guest kernel paths implicated in the report: hv_balloon free page reporting, user page faults, and the kernel-side do_swap_page trace. Compare behavior with balloon reporting disabled and collect the offered per-incident traces, especially host-side Hyper-V ETW during a stuck store. Done means identifying and fixing the cause so guest stores complete or fault and recover under memory pressure.

索引モデルが issue の本文から書いたものです。

説明

emailed-logs hypervisor-platform wsl2
Windows Version

10.0.26300.9818 (Snapdragon X2 Elite, ARM64, 48 GB RAM)

WSL Version

3.0.5.0 (also seen on 2.7.11 before upgrading)

Are you using WSL 1 or WSL 2?
  • WSL 2
Kernel Version

6.18.58.1-microsoft-standard-WSL2 (aarch64, PREEMPT_DYNAMIC)

Distro Version

Ubuntu 26.04 LTS

Other Software

Docker Desktop (engine 29.9.2, WSL integration), Node.js tools (Claude Code, MCP servers), PostgreSQL/MariaDB/PHP containers.

.wslconfig:

[wsl2]
memory=20GB
processors=12
swap=8GB
kernelCommandLine=hugepages=1300 preempt=full

[experimental]
autoMemoryReclaim=gradual
sparseVhd=true
Repro Steps

No deterministic repro. It happens several times a day on a busy dev VM (Docker containers, many Node processes, memory pressure that makes the balloon / free page reporting active):

  1. ARM64 Windows host, WSL 2 with hv_balloon loaded (default) and free page reporting active (autoMemoryReclaim=gradual, or the default page_reporting_order).
  2. Run a normal multi-process workload for a few hours (Docker containers, Node, Postgres).
  3. Sooner or later a thread sits at 100% CPU forever on a single store instruction.
Expected Behavior

A user-space store to a mapped, writable page completes, or faults and gets resolved by the guest kernel.

Actual Behavior

A user-space thread gets stuck forever on one store instruction.

  • The thread spins at 100% user time with zero voluntary context switches, and its PC never changes. We sampled it with gdb: two reads 2 s apart, over minutes to hours.

  • The instruction is always a write to memory:

    When (UTC-3) Process Instruction at stuck PC
    2026-10-31 postgres backends, soketi str / stp
    2026-11-04 14:03 node (JIT worker) str q3, [x0] inside memcpy, 10 min
    2026-11-04 (afternoon) node (hooks thread) dc zva inside memset, ~2 h
    2026-11-04 (afternoon) codex (Rust, tokio worker) ldadd
    2026-11-04 16:54 node str q3, [x0]
    2026-11-04 18:39 node (MCP server) str x3, [x2]
    2026-11-04 18:41 same process, V8 worker cas x0, x1, [x2]
    2026-11-04 18:42 another node stur w2, [x1, #19]
  • The page is never resolved for that thread. If we write the same byte that is already at the target address through /proc/PID/mem, the kernel resolves the page from outside (access_remote_vm). The stuck thread then continues in under 3 s: state goes from R to S and voluntary switches start to count again. We now do this automatically and it has worked in every case above:

    python3 -I -c "f=open('/proc/$PID/mem','r+b',buffering=0); f.seek($ADDR); b=f.read(1); f.seek($ADDR); f.write(b)"
    

    To us this points to a stale or lost stage-2 mapping or a permission state in the hypervisor for a guest page. The guest believes the page is present and writable, so it never faults it in again. A host-side write path re-establishes it. In our history the problem correlates with hv_balloon free page reporting (pages reported to the host and later reused by the guest):

    • With initcall_blacklist=init_balloon_drv (balloon disabled, 2026-11-01 18:29 to 2026-11-03 10:51, ~40 h) we saw no stuck store. The only hot threads then were ordinary user-space loops with a changing PC. With the balloon back on, the first kernel-side case came 11.5 h later.
    • With page_reporting.page_reporting_order=10 it still happened.
    • Upgrading from WSL 2.7.11 to 3.0.5 did not change it.

Kernel-side variant. The same thing can happen to a store done by the kernel on behalf of a task. On 2026-11-03 22:23:

  • A task's stime grew by ~100 ticks per second and its utime stayed constant.
  • The task was a Node V8 worker thread on CPU 4. The sysrq-l NMI backtrace showed do_swap_page → folio_add_new_anon_rmap → mod_memcg_lruvec_state+0x18c (inside memcg_rstat_updated, preemption disabled, a per-cpu ldadd).
  • With the default PREEMPT(none), that CPU never released the migration stopper. Every later sched_setaffinity / migration (affine_move_task) piled up behind it.
  • Load reached 458, processes became unkillable (SIGKILL ignored), and only wsl --shutdown recovered the VM.
  • Booting with preempt=full contains it: one CPU is lost instead of the whole VM. The root cause is still there.
Diagnostic Logs

We keep per-incident dossiers (gdb disassembly around the PC, registers, /proc/PID/task/*/stat|status, sysrq-l/sysrq-w traces, /proc/interrupts, free, cmdline) and can attach them or run collect-wsl-logs.ps1 during the next occurrence if that helps. Please tell us which traces would be most useful, for example Hyper-V ETW providers on the host while a thread is stuck.

Workaround for others hitting this on ARM64: either disable the balloon (kernelCommandLine=initcall_blacklist=init_balloon_drv, at the cost of memory never returning to Windows), or detect threads stuck on a store and rewrite the same byte through /proc/PID/mem as above, plus preempt=full so a kernel-side occurrence doesn't freeze the whole VM.

主要言語
C++
スター
34.4k
フォーク
1.9k
平均マージ
1日 23時間
マージ済み PR(30日)
166

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

microsoft/WSL のほかの issue

microsoft/WSL の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。