Mirrored networking: host-only 127.0.0.1 TCP traffic spins all cores in `Fse.sys!FsePortPoolGetPartitionsForPortUncached` (exclusive spin lock), freezing the desktop
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 5/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- cpp
调研方向
Start by building and running the main.go repro in mirrored mode (fserepro.exe load -dur 10s -rounds 10 -idle 4000) with the probe mode running alongside, then capture a wpr -start CPU trace and inspect the ExpWaitForSpinLockExclusiveAndAcquire samples under FsePortPoolGetPartitionsForPortUncached. From there, find the port-pool partition lookup and its cache in the fse.sys sources in this repo and read how reservations are added and removed on connection churn. Done means the uncached path no longer takes the pool-wide exclusive spin lock per loopback packet, or the cache stops going cold under churn, verified by the repro's kernel CPU per connection dropping back to the WSL-stopped baseline.
由索引模型根据 Issue 内容生成。
描述
Windows Version
Microsoft Windows [Version 10.0.26300.9818] (Windows 11 Home, 26H2)
WSL Version
3.0.3.0
Are you using WSL 1, WSL 2, or WSLC?
WSL 2
Kernel Version
6.18.54.1-1
Distro Version
Ubuntu 24.04
Other Software
.wslconfig:[wsl2] networkingMode=mirrored dnsTunneling=true autoProxy=true processors=16- Driver versions:
fse.sys10.0.26100.9818,tcpip.sys10.0.26100.8875,afd.sys10.0.26100.9949 - CPU: AMD Ryzen 9 7950X (16 cores / 32 threads), 96 GB RAM
- Other network-related software is installed (VMware Workstation, Tailscale, a wintun-based TUN client). None of their drivers appear in the hot call stacks below, and the problem goes away with only
wsl --shutdown.
Repro Steps
The traffic in this repro is entirely between Windows processes on the host (127.0.0.1 → 127.0.0.1 inside one Windows process). Nothing is sent to or from the Linux VM. The only variable is whether a mirrored-mode WSL VM is running.
- Set
networkingMode=mirroredand start a distro, keeping the VM up:wsl -d Ubuntu-24.04 -e sleep 600 - On the Windows host, build the program below (Go 1.27):
go build -o fserepro.exe main.go - Run it.
-idle 4000holds 4000 idle loopback connections open, then 32 workers repeatedly connect, do 10 round trips of 4 KB, and close:
In another window,fserepro.exe load -dur 10s -rounds 10 -idle 4000fserepro.exe probe -dur 14smeasures how late a 100 ms sleep wakes up. - Run
wsl --shutdownand repeat step 3.
main.go (standalone, standard library only)
// fserepro drives high-rate TCP traffic over 127.0.0.1 between two endpoints
// in one Windows process, and reports throughput and the user/kernel CPU split.
// The "probe" mode measures how late a 100 ms sleep wakes up, as a proxy for
// system responsiveness while the load runs.
package main
import (
"flag"
"fmt"
"io"
"net"
"os"
"runtime"
"sync"
"sync/atomic"
"syscall"
"time"
)
func cpuTimes() (user, kernel time.Duration) {
var c, e, k, u syscall.Filetime
h, _ := syscall.GetCurrentProcess()
if err := syscall.GetProcessTimes(h, &c, &e, &k, &u); err != nil {
return 0, 0
}
ft := func(f syscall.Filetime) time.Duration {
return time.Duration((int64(f.HighDateTime)<<32 | int64(f.LowDateTime)) * 100)
}
return ft(u), ft(k)
}
func lowerPriority() {
const belowNormal = 0x4000
k := syscall.NewLazyDLL("kernel32.dll")
h, _ := syscall.GetCurrentProcess()
k.NewProc("SetPriorityClass").Call(uintptr(h), belowNormal)
}
func echo(ln net.Listener) {
for {
c, err := ln.Accept()
if err != nil {
return
}
go func() {
defer c.Close()
io.Copy(c, c)
}()
}
}
// load runs workers that each open a connection, do `rounds` request/response
// exchanges of `payload` bytes, and close it. rounds <= 0 keeps one connection
// per worker for the whole run. Before the load starts, `idle` extra
// connections are opened and held without traffic, so that many local ports
// are in use while the workers run.
func load(workers int, dur time.Duration, payload, rounds, idle int) {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
fmt.Println("listen:", err)
os.Exit(1)
}
go echo(ln)
addr := ln.Addr().String()
held := make([]net.Conn, 0, idle)
for len(held) < idle {
c, err := net.Dial("tcp", addr)
if err != nil {
fmt.Printf("idle dial stopped at %d: %v\n", len(held), err)
break
}
held = append(held, c)
}
var trips, conns, errs atomic.Int64
deadline := time.Now().Add(dur)
u0, k0 := cpuTimes()
start := time.Now()
var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
out, in := make([]byte, payload), make([]byte, payload)
for time.Now().Before(deadline) {
c, err := net.Dial("tcp", addr)
if err != nil {
errs.Add(1)
time.Sleep(10 * time.Millisecond)
continue
}
c.(*net.TCPConn).SetLinger(0)
conns.Add(1)
for r := 0; (rounds <= 0 || r < rounds) && time.Now().Before(deadline); r++ {
if _, err := c.Write(out); err != nil {
errs.Add(1)
break
}
if _, err := io.ReadFull(c, in); err != nil {
errs.Add(1)
break
}
trips.Add(1)
}
c.Close()
}
}()
}
wg.Wait()
el := time.Since(start)
u1, k1 := cpuTimes()
ln.Close()
for _, c := range held {
c.Close()
}
u, k := u1-u0, k1-k0
fmt.Printf("workers=%d gomaxprocs=%d elapsed=%s payload=%d rounds=%d idle_conns=%d\n",
workers, runtime.GOMAXPROCS(0), el.Round(time.Millisecond), payload, rounds, len(held))
fmt.Printf("connections=%d (%.0f/s) round_trips=%d (%.0f/s) errors=%d\n",
conns.Load(), float64(conns.Load())/el.Seconds(), trips.Load(), float64(trips.Load())/el.Seconds(), errs.Load())
fmt.Printf("cpu_user=%.1fs cpu_kernel=%.1fs kernel_share=%.1f%%\n",
u.Seconds(), k.Seconds(), 100*k.Seconds()/(u+k).Seconds())
}
func probe(dur time.Duration) {
const tick = 100 * time.Millisecond
var worst time.Duration
var n, late int
end := time.Now().Add(dur)
for time.Now().Before(end) {
t := time.Now()
time.Sleep(tick)
over := time.Since(t) - tick
n++
if over > worst {
worst = over
}
if over > 200*time.Millisecond {
late++
}
}
fmt.Printf("probe samples=%d max_wakeup_delay=%s delays_over_200ms=%d\n", n, worst.Round(time.Millisecond), late)
}
func main() {
if len(os.Args) < 2 {
fmt.Println("usage: fserepro load|probe [flags]")
os.Exit(2)
}
fs := flag.NewFlagSet(os.Args[1], flag.ExitOnError)
workers := fs.Int("workers", runtime.NumCPU(), "concurrent client connections")
dur := fs.Duration("dur", 15*time.Second, "run time")
payload := fs.Int("payload", 4096, "bytes per request")
rounds := fs.Int("rounds", 0, "round trips per connection, 0 = one connection per worker")
idle := fs.Int("idle", 0, "extra idle connections held open during the load")
low := fs.Bool("belownormal", false, "run at below-normal priority")
fs.Parse(os.Args[2:])
if *low {
lowerPriority()
}
switch os.Args[1] {
case "load":
load(*workers, *dur, *payload, *rounds, *idle)
case "probe":
probe(*dur)
}
}
Expected Behavior
Host-only loopback TCP performance should not depend on whether a mirrored-mode WSL VM is running. These packets never leave the Windows host.
Actual Behavior
While the VM is running, the host loopback TCP path becomes much more expensive. The cost grows with the number of local ports in use. Connection-heavy workloads end up with every core spinning on a kernel spin lock.
Minimal repro results. Each run is 10–15 s with 32 workers on 32 hardware threads, load at below-normal priority:
| Scenario | WSL stopped | WSL running (mirrored) |
|---|---|---|
| 32 persistent connections, 0 idle | 271k round trips/s | 214k round trips/s |
| 32 persistent connections, 2000 idle | 251k round trips/s, probe max delay 1 ms | 137k round trips/s, probe max delay 23 ms |
| 32 persistent connections, ~8000 idle | 222k round trips/s, 3 ms | 141k round trips/s, 221 ms |
| connect + 10 round trips + close, 0 idle | 7,287 connections/s | 5,277 connections/s |
| connect + 10 round trips + close, 4000 idle | 1,015 connections/s, 19 s kernel CPU in 10 s | 118 connections/s, 224 s kernel CPU in 10 s (≈22 cores busy, 99.5% kernel) |
In the last row, kernel CPU per connection goes from about 1.9 ms to about 188 ms, roughly 100×.
Real-world impact: this was first noticed with a Go test suite that starts dozens of local proxy listeners and opens many concurrent loopback connections (about 500 established and about 5,000 in TIME_WAIT). While the WSL VM was running:
- The entire desktop froze for about 6.5 minutes. A 1-second performance-counter sampler only got scheduled every ~30 s.
- The test process used 118 s of user CPU and 3,398 s of kernel CPU.
- System totals: 85–100% privileged time, DPC + interrupt time under 3%, processor queue length up to 42.
- CPU clocks stayed normal at 5.2–5.7 GHz, with no thermal or PROCHOT throttling.
After wsl --shutdown, the same suite finished in 62 s instead of 396 s, used 27 s of kernel CPU, and the desktop stayed responsive.
The vSwitch (FSE Switch) and vEthernet (FSE HostVnic) adapters stay up after wsl --shutdown, but the contention disappears. It seems to depend on the per-port partition or reservation state that exists while the VM is running.
Hot path. Taken from a WPR CPU sampling trace of the minimal repro, symbolized with public symbols (xperf butterfly, restricted to samples in ExpWaitForSpinLock*):
afd!AfdFastConnectionSend 88.8%
tcpip!TcpTcbSend → tcpip!IppSendDatagramsCommon
tcpip!IppLoopbackEnqueue → tcpip!IppLbTransmitStackCallout → tcpip!IppLbIndicatePackets 95.7%
Fse!FseIpsClientReceivePackets → Fse!FseDemuxShimProcessInboundNblChain
Fse!FseShimsProcessSteerNblChain
Fse!FsePortPoolGetPartitionsForPort 95.7%
├─ nt!ExAcquireSpinLockShared → nt!ExpWaitForSpinLockSharedAndAcquire 9.9%
└─ Fse!FsePortPoolGetPartitionsForPortUncached 85.8%
nt!ExAcquireSpinLockExclusiveAtDpcLevel → nt!ExpWaitForSpinLockExclusiveAndAcquire 85.8%
Smaller contributors: the bind path, tcpip!InetAcquireWildcardPortUnderLock → PtLocalClientReservePortRange → Fse!FseLocalServerAcquirePorts → FsePortTrackerAcquirePorts → FsePartitionFindById (shared lock, 3%). Among non-waiting on-CPU time, the top fse.sys frame is Fse!FsePortPoolForEachReservationThatContainsPort, at about 3–5% of the process samples.
The real-world trace shows the same picture. 94% of the test process's samples are in ExpWaitForSpinLock{Exclusive,Shared}AndAcquire, entered through the same FseShimsProcessSteerNblChain → FsePortPoolGetPartitionsForPortUncached path.
Possible explanation (inferred from function names only): every loopback packet delivered by IppLbIndicatePackets is steered by FSE, which looks up the port's partitions. A cache miss falls through to ...Uncached, which takes a port-pool-wide spin lock exclusively at DISPATCH_LEVEL and appears to walk the reservations that contain the port. That walk would explain why the cost grows with the number of ports in use. Connection churn adds and removes reservations, which presumably keeps the cache cold. Because the waiters spin at DISPATCH_LEVEL, thread priority does not help, and other threads on those cores, including UI threads, are starved.
Workarounds: wsl --shutdown before running loopback-heavy workloads, switching to NAT mode, or limiting concurrency.
Diagnostic Logs
No logs attached. The standalone program above reproduces the problem in under a minute, and the call stacks above come from a WPR CPU sampling trace (wpr -start CPU) of that program. If needed, I can collect collect-wsl-logs.ps1 -LogProfile networking or a CPU trace during the repro and share it privately.
- 主要语言
- C++
- 星标
- 34.4k
- 派生
- 1.9k
- 平均合并
- 1 天 23 小时
- 30 天内合并 PR
- 166
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
microsoft/WSL 的其他 Issue
-
feature wsl2
难度 2/5 1-3 小时 新手友好度 80/100
维护者通常 1 天内回复
-
feature
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
feature kernel wsl2
难度 2/5 1-3 小时 新手友好度 68/100
microsoft/WSL#41107 · 3 条评论 · 1 个 reaction ·
维护者通常 1 天内回复
-
feature kernel
难度 2/5 1-3 小时 新手友好度 82/100
microsoft/WSL#41054 · 9 条评论 · 1 个 reaction ·
维护者通常 1 天内回复
-
feature GPU
难度 1/5 1-3 小时 新手友好度 70/100
维护者通常 1 天内回复
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 82/100
HarperFast/rocksdb-js#1131 ·
维护者通常 1 天内回复
-
area/daemon bug good first issue platform/linux priority/p2
难度 2/5 1-3 小时 新手友好度 72/100
ToshiNamura/Deimos#5 ·
-
needs-format
难度 2/5 1-3 小时 新手友好度 82/100
Niko1221/Strata#6251 · 1 条评论 ·
维护者通常 5 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 65/100
metasequoiaime/msime#11666 ·
维护者通常 1 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 70/100
patrick-hudson/put.gd-desktop#1679 · 1 条评论 ·
维护者通常 1 天内回复