Mirrored networking: host-only 127.0.0.1 TCP traffic spins all cores in `Fse.sys!FsePortPoolGetPartitionsForPortUncached` (exclusive spin lock), freezing the desktop

オープン
#42,282 コメント 5 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
5/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
cpp

調査の方向性

Start by building and running the main.go repro in mirrored mode (fserepro.exe load -dur 10s -rounds 10 -idle 4000) with the probe mode running alongside, then capture a wpr -start CPU trace and inspect the ExpWaitForSpinLockExclusiveAndAcquire samples under FsePortPoolGetPartitionsForPortUncached. From there, find the port-pool partition lookup and its cache in the fse.sys sources in this repo and read how reservations are added and removed on connection churn. Done means the uncached path no longer takes the pool-wide exclusive spin lock per loopback packet, or the cache stops going cold under churn, verified by the repro's kernel CPU per connection dropping back to the WSL-stopped baseline.

索引モデルが issue の本文から書いたものです。

説明

emailed-logs network wsl2
Windows Version

Microsoft Windows [Version 10.0.26300.9818] (Windows 11 Home, 26H2)

WSL Version

3.0.3.0

Are you using WSL 1, WSL 2, or WSLC?

WSL 2

Kernel Version

6.18.54.1-1

Distro Version

Ubuntu 24.04

Other Software
  • .wslconfig:
    [wsl2]
    networkingMode=mirrored
    dnsTunneling=true
    autoProxy=true
    processors=16
    
  • Driver versions: fse.sys 10.0.26100.9818, tcpip.sys 10.0.26100.8875, afd.sys 10.0.26100.9949
  • CPU: AMD Ryzen 9 7950X (16 cores / 32 threads), 96 GB RAM
  • Other network-related software is installed (VMware Workstation, Tailscale, a wintun-based TUN client). None of their drivers appear in the hot call stacks below, and the problem goes away with only wsl --shutdown.
Repro Steps

The traffic in this repro is entirely between Windows processes on the host (127.0.0.1 → 127.0.0.1 inside one Windows process). Nothing is sent to or from the Linux VM. The only variable is whether a mirrored-mode WSL VM is running.

  1. Set networkingMode=mirrored and start a distro, keeping the VM up: wsl -d Ubuntu-24.04 -e sleep 600
  2. On the Windows host, build the program below (Go 1.27): go build -o fserepro.exe main.go
  3. Run it. -idle 4000 holds 4000 idle loopback connections open, then 32 workers repeatedly connect, do 10 round trips of 4 KB, and close:
    fserepro.exe load -dur 10s -rounds 10 -idle 4000
    
    In another window, fserepro.exe probe -dur 14s measures how late a 100 ms sleep wakes up.
  4. Run wsl --shutdown and repeat step 3.
main.go (standalone, standard library only)
// fserepro drives high-rate TCP traffic over 127.0.0.1 between two endpoints
// in one Windows process, and reports throughput and the user/kernel CPU split.
// The "probe" mode measures how late a 100 ms sleep wakes up, as a proxy for
// system responsiveness while the load runs.
package main

import (
	"flag"
	"fmt"
	"io"
	"net"
	"os"
	"runtime"
	"sync"
	"sync/atomic"
	"syscall"
	"time"
)

func cpuTimes() (user, kernel time.Duration) {
	var c, e, k, u syscall.Filetime
	h, _ := syscall.GetCurrentProcess()
	if err := syscall.GetProcessTimes(h, &c, &e, &k, &u); err != nil {
		return 0, 0
	}
	ft := func(f syscall.Filetime) time.Duration {
		return time.Duration((int64(f.HighDateTime)<<32 | int64(f.LowDateTime)) * 100)
	}
	return ft(u), ft(k)
}

func lowerPriority() {
	const belowNormal = 0x4000
	k := syscall.NewLazyDLL("kernel32.dll")
	h, _ := syscall.GetCurrentProcess()
	k.NewProc("SetPriorityClass").Call(uintptr(h), belowNormal)
}

func echo(ln net.Listener) {
	for {
		c, err := ln.Accept()
		if err != nil {
			return
		}
		go func() {
			defer c.Close()
			io.Copy(c, c)
		}()
	}
}

// load runs workers that each open a connection, do `rounds` request/response
// exchanges of `payload` bytes, and close it. rounds <= 0 keeps one connection
// per worker for the whole run. Before the load starts, `idle` extra
// connections are opened and held without traffic, so that many local ports
// are in use while the workers run.
func load(workers int, dur time.Duration, payload, rounds, idle int) {
	ln, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		fmt.Println("listen:", err)
		os.Exit(1)
	}
	go echo(ln)
	addr := ln.Addr().String()
	held := make([]net.Conn, 0, idle)
	for len(held) < idle {
		c, err := net.Dial("tcp", addr)
		if err != nil {
			fmt.Printf("idle dial stopped at %d: %v\n", len(held), err)
			break
		}
		held = append(held, c)
	}
	var trips, conns, errs atomic.Int64
	deadline := time.Now().Add(dur)
	u0, k0 := cpuTimes()
	start := time.Now()
	var wg sync.WaitGroup
	for i := 0; i < workers; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			out, in := make([]byte, payload), make([]byte, payload)
			for time.Now().Before(deadline) {
				c, err := net.Dial("tcp", addr)
				if err != nil {
					errs.Add(1)
					time.Sleep(10 * time.Millisecond)
					continue
				}
				c.(*net.TCPConn).SetLinger(0)
				conns.Add(1)
				for r := 0; (rounds <= 0 || r < rounds) && time.Now().Before(deadline); r++ {
					if _, err := c.Write(out); err != nil {
						errs.Add(1)
						break
					}
					if _, err := io.ReadFull(c, in); err != nil {
						errs.Add(1)
						break
					}
					trips.Add(1)
				}
				c.Close()
			}
		}()
	}
	wg.Wait()
	el := time.Since(start)
	u1, k1 := cpuTimes()
	ln.Close()
	for _, c := range held {
		c.Close()
	}
	u, k := u1-u0, k1-k0
	fmt.Printf("workers=%d gomaxprocs=%d elapsed=%s payload=%d rounds=%d idle_conns=%d\n",
		workers, runtime.GOMAXPROCS(0), el.Round(time.Millisecond), payload, rounds, len(held))
	fmt.Printf("connections=%d (%.0f/s) round_trips=%d (%.0f/s) errors=%d\n",
		conns.Load(), float64(conns.Load())/el.Seconds(), trips.Load(), float64(trips.Load())/el.Seconds(), errs.Load())
	fmt.Printf("cpu_user=%.1fs cpu_kernel=%.1fs kernel_share=%.1f%%\n",
		u.Seconds(), k.Seconds(), 100*k.Seconds()/(u+k).Seconds())
}

func probe(dur time.Duration) {
	const tick = 100 * time.Millisecond
	var worst time.Duration
	var n, late int
	end := time.Now().Add(dur)
	for time.Now().Before(end) {
		t := time.Now()
		time.Sleep(tick)
		over := time.Since(t) - tick
		n++
		if over > worst {
			worst = over
		}
		if over > 200*time.Millisecond {
			late++
		}
	}
	fmt.Printf("probe samples=%d max_wakeup_delay=%s delays_over_200ms=%d\n", n, worst.Round(time.Millisecond), late)
}

func main() {
	if len(os.Args) < 2 {
		fmt.Println("usage: fserepro load|probe [flags]")
		os.Exit(2)
	}
	fs := flag.NewFlagSet(os.Args[1], flag.ExitOnError)
	workers := fs.Int("workers", runtime.NumCPU(), "concurrent client connections")
	dur := fs.Duration("dur", 15*time.Second, "run time")
	payload := fs.Int("payload", 4096, "bytes per request")
	rounds := fs.Int("rounds", 0, "round trips per connection, 0 = one connection per worker")
	idle := fs.Int("idle", 0, "extra idle connections held open during the load")
	low := fs.Bool("belownormal", false, "run at below-normal priority")
	fs.Parse(os.Args[2:])
	if *low {
		lowerPriority()
	}
	switch os.Args[1] {
	case "load":
		load(*workers, *dur, *payload, *rounds, *idle)
	case "probe":
		probe(*dur)
	}
}
Expected Behavior

Host-only loopback TCP performance should not depend on whether a mirrored-mode WSL VM is running. These packets never leave the Windows host.

Actual Behavior

While the VM is running, the host loopback TCP path becomes much more expensive. The cost grows with the number of local ports in use. Connection-heavy workloads end up with every core spinning on a kernel spin lock.

Minimal repro results. Each run is 10–15 s with 32 workers on 32 hardware threads, load at below-normal priority:

Scenario WSL stopped WSL running (mirrored)
32 persistent connections, 0 idle 271k round trips/s 214k round trips/s
32 persistent connections, 2000 idle 251k round trips/s, probe max delay 1 ms 137k round trips/s, probe max delay 23 ms
32 persistent connections, ~8000 idle 222k round trips/s, 3 ms 141k round trips/s, 221 ms
connect + 10 round trips + close, 0 idle 7,287 connections/s 5,277 connections/s
connect + 10 round trips + close, 4000 idle 1,015 connections/s, 19 s kernel CPU in 10 s 118 connections/s, 224 s kernel CPU in 10 s (≈22 cores busy, 99.5% kernel)

In the last row, kernel CPU per connection goes from about 1.9 ms to about 188 ms, roughly 100×.

Real-world impact: this was first noticed with a Go test suite that starts dozens of local proxy listeners and opens many concurrent loopback connections (about 500 established and about 5,000 in TIME_WAIT). While the WSL VM was running:

  • The entire desktop froze for about 6.5 minutes. A 1-second performance-counter sampler only got scheduled every ~30 s.
  • The test process used 118 s of user CPU and 3,398 s of kernel CPU.
  • System totals: 85–100% privileged time, DPC + interrupt time under 3%, processor queue length up to 42.
  • CPU clocks stayed normal at 5.2–5.7 GHz, with no thermal or PROCHOT throttling.

After wsl --shutdown, the same suite finished in 62 s instead of 396 s, used 27 s of kernel CPU, and the desktop stayed responsive.

The vSwitch (FSE Switch) and vEthernet (FSE HostVnic) adapters stay up after wsl --shutdown, but the contention disappears. It seems to depend on the per-port partition or reservation state that exists while the VM is running.

Hot path. Taken from a WPR CPU sampling trace of the minimal repro, symbolized with public symbols (xperf butterfly, restricted to samples in ExpWaitForSpinLock*):

afd!AfdFastConnectionSend                                  88.8%
 tcpip!TcpTcbSend → tcpip!IppSendDatagramsCommon
  tcpip!IppLoopbackEnqueue → tcpip!IppLbTransmitStackCallout → tcpip!IppLbIndicatePackets   95.7%
   Fse!FseIpsClientReceivePackets → Fse!FseDemuxShimProcessInboundNblChain
    Fse!FseShimsProcessSteerNblChain
     Fse!FsePortPoolGetPartitionsForPort                    95.7%
      ├─ nt!ExAcquireSpinLockShared → nt!ExpWaitForSpinLockSharedAndAcquire                  9.9%
      └─ Fse!FsePortPoolGetPartitionsForPortUncached        85.8%
          nt!ExAcquireSpinLockExclusiveAtDpcLevel → nt!ExpWaitForSpinLockExclusiveAndAcquire 85.8%

Smaller contributors: the bind path, tcpip!InetAcquireWildcardPortUnderLock → PtLocalClientReservePortRange → Fse!FseLocalServerAcquirePorts → FsePortTrackerAcquirePorts → FsePartitionFindById (shared lock, 3%). Among non-waiting on-CPU time, the top fse.sys frame is Fse!FsePortPoolForEachReservationThatContainsPort, at about 3–5% of the process samples.

The real-world trace shows the same picture. 94% of the test process's samples are in ExpWaitForSpinLock{Exclusive,Shared}AndAcquire, entered through the same FseShimsProcessSteerNblChain → FsePortPoolGetPartitionsForPortUncached path.

Possible explanation (inferred from function names only): every loopback packet delivered by IppLbIndicatePackets is steered by FSE, which looks up the port's partitions. A cache miss falls through to ...Uncached, which takes a port-pool-wide spin lock exclusively at DISPATCH_LEVEL and appears to walk the reservations that contain the port. That walk would explain why the cost grows with the number of ports in use. Connection churn adds and removes reservations, which presumably keeps the cache cold. Because the waiters spin at DISPATCH_LEVEL, thread priority does not help, and other threads on those cores, including UI threads, are starved.

Workarounds: wsl --shutdown before running loopback-heavy workloads, switching to NAT mode, or limiting concurrency.

Diagnostic Logs

No logs attached. The standalone program above reproduces the problem in under a minute, and the call stacks above come from a WPR CPU sampling trace (wpr -start CPU) of that program. If needed, I can collect collect-wsl-logs.ps1 -LogProfile networking or a CPU trace during the repro and share it privately.

主要言語
C++
スター
34.4k
フォーク
1.9k
平均マージ
1日 23時間
マージ済み PR(30日)
166

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

microsoft/WSL のほかの issue

microsoft/WSL の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。