Mirrored networking: host-only 127.0.0.1 TCP traffic spins all cores in `Fse.sys!FsePortPoolGetPartitionsForPortUncached` (exclusive spin lock), freezing the desktop

未关闭
#42,282 5 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
5/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
cpp

调研方向

Start by building and running the main.go repro in mirrored mode (fserepro.exe load -dur 10s -rounds 10 -idle 4000) with the probe mode running alongside, then capture a wpr -start CPU trace and inspect the ExpWaitForSpinLockExclusiveAndAcquire samples under FsePortPoolGetPartitionsForPortUncached. From there, find the port-pool partition lookup and its cache in the fse.sys sources in this repo and read how reservations are added and removed on connection churn. Done means the uncached path no longer takes the pool-wide exclusive spin lock per loopback packet, or the cache stops going cold under churn, verified by the repro's kernel CPU per connection dropping back to the WSL-stopped baseline.

由索引模型根据 Issue 内容生成。

描述

emailed-logs network wsl2
Windows Version

Microsoft Windows [Version 10.0.26300.9818] (Windows 11 Home, 26H2)

WSL Version

3.0.3.0

Are you using WSL 1, WSL 2, or WSLC?

WSL 2

Kernel Version

6.18.54.1-1

Distro Version

Ubuntu 24.04

Other Software
  • .wslconfig:
    [wsl2]
    networkingMode=mirrored
    dnsTunneling=true
    autoProxy=true
    processors=16
    
  • Driver versions: fse.sys 10.0.26100.9818, tcpip.sys 10.0.26100.8875, afd.sys 10.0.26100.9949
  • CPU: AMD Ryzen 9 7950X (16 cores / 32 threads), 96 GB RAM
  • Other network-related software is installed (VMware Workstation, Tailscale, a wintun-based TUN client). None of their drivers appear in the hot call stacks below, and the problem goes away with only wsl --shutdown.
Repro Steps

The traffic in this repro is entirely between Windows processes on the host (127.0.0.1 → 127.0.0.1 inside one Windows process). Nothing is sent to or from the Linux VM. The only variable is whether a mirrored-mode WSL VM is running.

  1. Set networkingMode=mirrored and start a distro, keeping the VM up: wsl -d Ubuntu-24.04 -e sleep 600
  2. On the Windows host, build the program below (Go 1.27): go build -o fserepro.exe main.go
  3. Run it. -idle 4000 holds 4000 idle loopback connections open, then 32 workers repeatedly connect, do 10 round trips of 4 KB, and close:
    fserepro.exe load -dur 10s -rounds 10 -idle 4000
    
    In another window, fserepro.exe probe -dur 14s measures how late a 100 ms sleep wakes up.
  4. Run wsl --shutdown and repeat step 3.
main.go (standalone, standard library only)
// fserepro drives high-rate TCP traffic over 127.0.0.1 between two endpoints
// in one Windows process, and reports throughput and the user/kernel CPU split.
// The "probe" mode measures how late a 100 ms sleep wakes up, as a proxy for
// system responsiveness while the load runs.
package main

import (
	"flag"
	"fmt"
	"io"
	"net"
	"os"
	"runtime"
	"sync"
	"sync/atomic"
	"syscall"
	"time"
)

func cpuTimes() (user, kernel time.Duration) {
	var c, e, k, u syscall.Filetime
	h, _ := syscall.GetCurrentProcess()
	if err := syscall.GetProcessTimes(h, &c, &e, &k, &u); err != nil {
		return 0, 0
	}
	ft := func(f syscall.Filetime) time.Duration {
		return time.Duration((int64(f.HighDateTime)<<32 | int64(f.LowDateTime)) * 100)
	}
	return ft(u), ft(k)
}

func lowerPriority() {
	const belowNormal = 0x4000
	k := syscall.NewLazyDLL("kernel32.dll")
	h, _ := syscall.GetCurrentProcess()
	k.NewProc("SetPriorityClass").Call(uintptr(h), belowNormal)
}

func echo(ln net.Listener) {
	for {
		c, err := ln.Accept()
		if err != nil {
			return
		}
		go func() {
			defer c.Close()
			io.Copy(c, c)
		}()
	}
}

// load runs workers that each open a connection, do `rounds` request/response
// exchanges of `payload` bytes, and close it. rounds <= 0 keeps one connection
// per worker for the whole run. Before the load starts, `idle` extra
// connections are opened and held without traffic, so that many local ports
// are in use while the workers run.
func load(workers int, dur time.Duration, payload, rounds, idle int) {
	ln, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		fmt.Println("listen:", err)
		os.Exit(1)
	}
	go echo(ln)
	addr := ln.Addr().String()
	held := make([]net.Conn, 0, idle)
	for len(held) < idle {
		c, err := net.Dial("tcp", addr)
		if err != nil {
			fmt.Printf("idle dial stopped at %d: %v\n", len(held), err)
			break
		}
		held = append(held, c)
	}
	var trips, conns, errs atomic.Int64
	deadline := time.Now().Add(dur)
	u0, k0 := cpuTimes()
	start := time.Now()
	var wg sync.WaitGroup
	for i := 0; i < workers; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			out, in := make([]byte, payload), make([]byte, payload)
			for time.Now().Before(deadline) {
				c, err := net.Dial("tcp", addr)
				if err != nil {
					errs.Add(1)
					time.Sleep(10 * time.Millisecond)
					continue
				}
				c.(*net.TCPConn).SetLinger(0)
				conns.Add(1)
				for r := 0; (rounds <= 0 || r < rounds) && time.Now().Before(deadline); r++ {
					if _, err := c.Write(out); err != nil {
						errs.Add(1)
						break
					}
					if _, err := io.ReadFull(c, in); err != nil {
						errs.Add(1)
						break
					}
					trips.Add(1)
				}
				c.Close()
			}
		}()
	}
	wg.Wait()
	el := time.Since(start)
	u1, k1 := cpuTimes()
	ln.Close()
	for _, c := range held {
		c.Close()
	}
	u, k := u1-u0, k1-k0
	fmt.Printf("workers=%d gomaxprocs=%d elapsed=%s payload=%d rounds=%d idle_conns=%d\n",
		workers, runtime.GOMAXPROCS(0), el.Round(time.Millisecond), payload, rounds, len(held))
	fmt.Printf("connections=%d (%.0f/s) round_trips=%d (%.0f/s) errors=%d\n",
		conns.Load(), float64(conns.Load())/el.Seconds(), trips.Load(), float64(trips.Load())/el.Seconds(), errs.Load())
	fmt.Printf("cpu_user=%.1fs cpu_kernel=%.1fs kernel_share=%.1f%%\n",
		u.Seconds(), k.Seconds(), 100*k.Seconds()/(u+k).Seconds())
}

func probe(dur time.Duration) {
	const tick = 100 * time.Millisecond
	var worst time.Duration
	var n, late int
	end := time.Now().Add(dur)
	for time.Now().Before(end) {
		t := time.Now()
		time.Sleep(tick)
		over := time.Since(t) - tick
		n++
		if over > worst {
			worst = over
		}
		if over > 200*time.Millisecond {
			late++
		}
	}
	fmt.Printf("probe samples=%d max_wakeup_delay=%s delays_over_200ms=%d\n", n, worst.Round(time.Millisecond), late)
}

func main() {
	if len(os.Args) < 2 {
		fmt.Println("usage: fserepro load|probe [flags]")
		os.Exit(2)
	}
	fs := flag.NewFlagSet(os.Args[1], flag.ExitOnError)
	workers := fs.Int("workers", runtime.NumCPU(), "concurrent client connections")
	dur := fs.Duration("dur", 15*time.Second, "run time")
	payload := fs.Int("payload", 4096, "bytes per request")
	rounds := fs.Int("rounds", 0, "round trips per connection, 0 = one connection per worker")
	idle := fs.Int("idle", 0, "extra idle connections held open during the load")
	low := fs.Bool("belownormal", false, "run at below-normal priority")
	fs.Parse(os.Args[2:])
	if *low {
		lowerPriority()
	}
	switch os.Args[1] {
	case "load":
		load(*workers, *dur, *payload, *rounds, *idle)
	case "probe":
		probe(*dur)
	}
}
Expected Behavior

Host-only loopback TCP performance should not depend on whether a mirrored-mode WSL VM is running. These packets never leave the Windows host.

Actual Behavior

While the VM is running, the host loopback TCP path becomes much more expensive. The cost grows with the number of local ports in use. Connection-heavy workloads end up with every core spinning on a kernel spin lock.

Minimal repro results. Each run is 10–15 s with 32 workers on 32 hardware threads, load at below-normal priority:

Scenario WSL stopped WSL running (mirrored)
32 persistent connections, 0 idle 271k round trips/s 214k round trips/s
32 persistent connections, 2000 idle 251k round trips/s, probe max delay 1 ms 137k round trips/s, probe max delay 23 ms
32 persistent connections, ~8000 idle 222k round trips/s, 3 ms 141k round trips/s, 221 ms
connect + 10 round trips + close, 0 idle 7,287 connections/s 5,277 connections/s
connect + 10 round trips + close, 4000 idle 1,015 connections/s, 19 s kernel CPU in 10 s 118 connections/s, 224 s kernel CPU in 10 s (≈22 cores busy, 99.5% kernel)

In the last row, kernel CPU per connection goes from about 1.9 ms to about 188 ms, roughly 100×.

Real-world impact: this was first noticed with a Go test suite that starts dozens of local proxy listeners and opens many concurrent loopback connections (about 500 established and about 5,000 in TIME_WAIT). While the WSL VM was running:

  • The entire desktop froze for about 6.5 minutes. A 1-second performance-counter sampler only got scheduled every ~30 s.
  • The test process used 118 s of user CPU and 3,398 s of kernel CPU.
  • System totals: 85–100% privileged time, DPC + interrupt time under 3%, processor queue length up to 42.
  • CPU clocks stayed normal at 5.2–5.7 GHz, with no thermal or PROCHOT throttling.

After wsl --shutdown, the same suite finished in 62 s instead of 396 s, used 27 s of kernel CPU, and the desktop stayed responsive.

The vSwitch (FSE Switch) and vEthernet (FSE HostVnic) adapters stay up after wsl --shutdown, but the contention disappears. It seems to depend on the per-port partition or reservation state that exists while the VM is running.

Hot path. Taken from a WPR CPU sampling trace of the minimal repro, symbolized with public symbols (xperf butterfly, restricted to samples in ExpWaitForSpinLock*):

afd!AfdFastConnectionSend                                  88.8%
 tcpip!TcpTcbSend → tcpip!IppSendDatagramsCommon
  tcpip!IppLoopbackEnqueue → tcpip!IppLbTransmitStackCallout → tcpip!IppLbIndicatePackets   95.7%
   Fse!FseIpsClientReceivePackets → Fse!FseDemuxShimProcessInboundNblChain
    Fse!FseShimsProcessSteerNblChain
     Fse!FsePortPoolGetPartitionsForPort                    95.7%
      ├─ nt!ExAcquireSpinLockShared → nt!ExpWaitForSpinLockSharedAndAcquire                  9.9%
      └─ Fse!FsePortPoolGetPartitionsForPortUncached        85.8%
          nt!ExAcquireSpinLockExclusiveAtDpcLevel → nt!ExpWaitForSpinLockExclusiveAndAcquire 85.8%

Smaller contributors: the bind path, tcpip!InetAcquireWildcardPortUnderLock → PtLocalClientReservePortRange → Fse!FseLocalServerAcquirePorts → FsePortTrackerAcquirePorts → FsePartitionFindById (shared lock, 3%). Among non-waiting on-CPU time, the top fse.sys frame is Fse!FsePortPoolForEachReservationThatContainsPort, at about 3–5% of the process samples.

The real-world trace shows the same picture. 94% of the test process's samples are in ExpWaitForSpinLock{Exclusive,Shared}AndAcquire, entered through the same FseShimsProcessSteerNblChain → FsePortPoolGetPartitionsForPortUncached path.

Possible explanation (inferred from function names only): every loopback packet delivered by IppLbIndicatePackets is steered by FSE, which looks up the port's partitions. A cache miss falls through to ...Uncached, which takes a port-pool-wide spin lock exclusively at DISPATCH_LEVEL and appears to walk the reservations that contain the port. That walk would explain why the cost grows with the number of ports in use. Connection churn adds and removes reservations, which presumably keeps the cache cold. Because the waiters spin at DISPATCH_LEVEL, thread priority does not help, and other threads on those cores, including UI threads, are starved.

Workarounds: wsl --shutdown before running loopback-heavy workloads, switching to NAT mode, or limiting concurrency.

Diagnostic Logs

No logs attached. The standalone program above reproduces the problem in under a minute, and the call stacks above come from a WPR CPU sampling trace (wpr -start CPU) of that program. If needed, I can collect collect-wsl-logs.ps1 -LogProfile networking or a CPU trace during the repro and share it privately.

主要语言
C++
星标
34.4k
派生
1.9k
平均合并
1 天 23 小时
30 天内合并 PR
166

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

microsoft/WSL 的其他 Issue

查看 microsoft/WSL 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。