Mirrored networking: host-only 127.0.0.1 TCP traffic spins all cores in `Fse.sys!FsePortPoolGetPartitionsForPortUncached` (exclusive spin lock), freezing the desktop
Maintainer thường phản hồi trong vòng 1 ngày
Chưa có ai nhận issue này.
Đánh giá
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức phù hợp với người mới
- 5/100
- Loại issue
- Lỗi
- Độ rõ ràng
- Khá rõ ràng
- Mức độ hoạt động
- Sôi nổi
- Công nghệ
- cpp
- Lĩnh vực
- networking, operating-systems, performance
Hướng nghiên cứu
Start by building and running the main.go repro in mirrored mode (fserepro.exe load -dur 10s -rounds 10 -idle 4000) with the probe mode running alongside, then capture a wpr -start CPU trace and inspect the ExpWaitForSpinLockExclusiveAndAcquire samples under FsePortPoolGetPartitionsForPortUncached. From there, find the port-pool partition lookup and its cache in the fse.sys sources in this repo and read how reservations are added and removed on connection churn. Done means the uncached path no longer takes the pool-wide exclusive spin lock per loopback packet, or the cache stops going cold under churn, verified by the repro's kernel CPU per connection dropping back to the WSL-stopped baseline.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Mô tả
Windows Version
Microsoft Windows [Version 10.0.26300.9818] (Windows 11 Home, 26H2)
WSL Version
3.0.3.0
Are you using WSL 1, WSL 2, or WSLC?
WSL 2
Kernel Version
6.18.54.1-1
Distro Version
Ubuntu 24.04
Other Software
.wslconfig:[wsl2] networkingMode=mirrored dnsTunneling=true autoProxy=true processors=16- Driver versions:
fse.sys10.0.26100.9818,tcpip.sys10.0.26100.8875,afd.sys10.0.26100.9949 - CPU: AMD Ryzen 9 7950X (16 cores / 32 threads), 96 GB RAM
- Other network-related software is installed (VMware Workstation, Tailscale, a wintun-based TUN client). None of their drivers appear in the hot call stacks below, and the problem goes away with only
wsl --shutdown.
Repro Steps
The traffic in this repro is entirely between Windows processes on the host (127.0.0.1 → 127.0.0.1 inside one Windows process). Nothing is sent to or from the Linux VM. The only variable is whether a mirrored-mode WSL VM is running.
- Set
networkingMode=mirroredand start a distro, keeping the VM up:wsl -d Ubuntu-24.04 -e sleep 600 - On the Windows host, build the program below (Go 1.27):
go build -o fserepro.exe main.go - Run it.
-idle 4000holds 4000 idle loopback connections open, then 32 workers repeatedly connect, do 10 round trips of 4 KB, and close:
In another window,fserepro.exe load -dur 10s -rounds 10 -idle 4000fserepro.exe probe -dur 14smeasures how late a 100 ms sleep wakes up. - Run
wsl --shutdownand repeat step 3.
main.go (standalone, standard library only)
// fserepro drives high-rate TCP traffic over 127.0.0.1 between two endpoints
// in one Windows process, and reports throughput and the user/kernel CPU split.
// The "probe" mode measures how late a 100 ms sleep wakes up, as a proxy for
// system responsiveness while the load runs.
package main
import (
"flag"
"fmt"
"io"
"net"
"os"
"runtime"
"sync"
"sync/atomic"
"syscall"
"time"
)
func cpuTimes() (user, kernel time.Duration) {
var c, e, k, u syscall.Filetime
h, _ := syscall.GetCurrentProcess()
if err := syscall.GetProcessTimes(h, &c, &e, &k, &u); err != nil {
return 0, 0
}
ft := func(f syscall.Filetime) time.Duration {
return time.Duration((int64(f.HighDateTime)<<32 | int64(f.LowDateTime)) * 100)
}
return ft(u), ft(k)
}
func lowerPriority() {
const belowNormal = 0x4000
k := syscall.NewLazyDLL("kernel32.dll")
h, _ := syscall.GetCurrentProcess()
k.NewProc("SetPriorityClass").Call(uintptr(h), belowNormal)
}
func echo(ln net.Listener) {
for {
c, err := ln.Accept()
if err != nil {
return
}
go func() {
defer c.Close()
io.Copy(c, c)
}()
}
}
// load runs workers that each open a connection, do `rounds` request/response
// exchanges of `payload` bytes, and close it. rounds <= 0 keeps one connection
// per worker for the whole run. Before the load starts, `idle` extra
// connections are opened and held without traffic, so that many local ports
// are in use while the workers run.
func load(workers int, dur time.Duration, payload, rounds, idle int) {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
fmt.Println("listen:", err)
os.Exit(1)
}
go echo(ln)
addr := ln.Addr().String()
held := make([]net.Conn, 0, idle)
for len(held) < idle {
c, err := net.Dial("tcp", addr)
if err != nil {
fmt.Printf("idle dial stopped at %d: %v\n", len(held), err)
break
}
held = append(held, c)
}
var trips, conns, errs atomic.Int64
deadline := time.Now().Add(dur)
u0, k0 := cpuTimes()
start := time.Now()
var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
out, in := make([]byte, payload), make([]byte, payload)
for time.Now().Before(deadline) {
c, err := net.Dial("tcp", addr)
if err != nil {
errs.Add(1)
time.Sleep(10 * time.Millisecond)
continue
}
c.(*net.TCPConn).SetLinger(0)
conns.Add(1)
for r := 0; (rounds <= 0 || r < rounds) && time.Now().Before(deadline); r++ {
if _, err := c.Write(out); err != nil {
errs.Add(1)
break
}
if _, err := io.ReadFull(c, in); err != nil {
errs.Add(1)
break
}
trips.Add(1)
}
c.Close()
}
}()
}
wg.Wait()
el := time.Since(start)
u1, k1 := cpuTimes()
ln.Close()
for _, c := range held {
c.Close()
}
u, k := u1-u0, k1-k0
fmt.Printf("workers=%d gomaxprocs=%d elapsed=%s payload=%d rounds=%d idle_conns=%d\n",
workers, runtime.GOMAXPROCS(0), el.Round(time.Millisecond), payload, rounds, len(held))
fmt.Printf("connections=%d (%.0f/s) round_trips=%d (%.0f/s) errors=%d\n",
conns.Load(), float64(conns.Load())/el.Seconds(), trips.Load(), float64(trips.Load())/el.Seconds(), errs.Load())
fmt.Printf("cpu_user=%.1fs cpu_kernel=%.1fs kernel_share=%.1f%%\n",
u.Seconds(), k.Seconds(), 100*k.Seconds()/(u+k).Seconds())
}
func probe(dur time.Duration) {
const tick = 100 * time.Millisecond
var worst time.Duration
var n, late int
end := time.Now().Add(dur)
for time.Now().Before(end) {
t := time.Now()
time.Sleep(tick)
over := time.Since(t) - tick
n++
if over > worst {
worst = over
}
if over > 200*time.Millisecond {
late++
}
}
fmt.Printf("probe samples=%d max_wakeup_delay=%s delays_over_200ms=%d\n", n, worst.Round(time.Millisecond), late)
}
func main() {
if len(os.Args) < 2 {
fmt.Println("usage: fserepro load|probe [flags]")
os.Exit(2)
}
fs := flag.NewFlagSet(os.Args[1], flag.ExitOnError)
workers := fs.Int("workers", runtime.NumCPU(), "concurrent client connections")
dur := fs.Duration("dur", 15*time.Second, "run time")
payload := fs.Int("payload", 4096, "bytes per request")
rounds := fs.Int("rounds", 0, "round trips per connection, 0 = one connection per worker")
idle := fs.Int("idle", 0, "extra idle connections held open during the load")
low := fs.Bool("belownormal", false, "run at below-normal priority")
fs.Parse(os.Args[2:])
if *low {
lowerPriority()
}
switch os.Args[1] {
case "load":
load(*workers, *dur, *payload, *rounds, *idle)
case "probe":
probe(*dur)
}
}
Expected Behavior
Host-only loopback TCP performance should not depend on whether a mirrored-mode WSL VM is running. These packets never leave the Windows host.
Actual Behavior
While the VM is running, the host loopback TCP path becomes much more expensive. The cost grows with the number of local ports in use. Connection-heavy workloads end up with every core spinning on a kernel spin lock.
Minimal repro results. Each run is 10–15 s with 32 workers on 32 hardware threads, load at below-normal priority:
| Scenario | WSL stopped | WSL running (mirrored) |
|---|---|---|
| 32 persistent connections, 0 idle | 271k round trips/s | 214k round trips/s |
| 32 persistent connections, 2000 idle | 251k round trips/s, probe max delay 1 ms | 137k round trips/s, probe max delay 23 ms |
| 32 persistent connections, ~8000 idle | 222k round trips/s, 3 ms | 141k round trips/s, 221 ms |
| connect + 10 round trips + close, 0 idle | 7,287 connections/s | 5,277 connections/s |
| connect + 10 round trips + close, 4000 idle | 1,015 connections/s, 19 s kernel CPU in 10 s | 118 connections/s, 224 s kernel CPU in 10 s (≈22 cores busy, 99.5% kernel) |
In the last row, kernel CPU per connection goes from about 1.9 ms to about 188 ms, roughly 100×.
Real-world impact: this was first noticed with a Go test suite that starts dozens of local proxy listeners and opens many concurrent loopback connections (about 500 established and about 5,000 in TIME_WAIT). While the WSL VM was running:
- The entire desktop froze for about 6.5 minutes. A 1-second performance-counter sampler only got scheduled every ~30 s.
- The test process used 118 s of user CPU and 3,398 s of kernel CPU.
- System totals: 85–100% privileged time, DPC + interrupt time under 3%, processor queue length up to 42.
- CPU clocks stayed normal at 5.2–5.7 GHz, with no thermal or PROCHOT throttling.
After wsl --shutdown, the same suite finished in 62 s instead of 396 s, used 27 s of kernel CPU, and the desktop stayed responsive.
The vSwitch (FSE Switch) and vEthernet (FSE HostVnic) adapters stay up after wsl --shutdown, but the contention disappears. It seems to depend on the per-port partition or reservation state that exists while the VM is running.
Hot path. Taken from a WPR CPU sampling trace of the minimal repro, symbolized with public symbols (xperf butterfly, restricted to samples in ExpWaitForSpinLock*):
afd!AfdFastConnectionSend 88.8%
tcpip!TcpTcbSend → tcpip!IppSendDatagramsCommon
tcpip!IppLoopbackEnqueue → tcpip!IppLbTransmitStackCallout → tcpip!IppLbIndicatePackets 95.7%
Fse!FseIpsClientReceivePackets → Fse!FseDemuxShimProcessInboundNblChain
Fse!FseShimsProcessSteerNblChain
Fse!FsePortPoolGetPartitionsForPort 95.7%
├─ nt!ExAcquireSpinLockShared → nt!ExpWaitForSpinLockSharedAndAcquire 9.9%
└─ Fse!FsePortPoolGetPartitionsForPortUncached 85.8%
nt!ExAcquireSpinLockExclusiveAtDpcLevel → nt!ExpWaitForSpinLockExclusiveAndAcquire 85.8%
Smaller contributors: the bind path, tcpip!InetAcquireWildcardPortUnderLock → PtLocalClientReservePortRange → Fse!FseLocalServerAcquirePorts → FsePortTrackerAcquirePorts → FsePartitionFindById (shared lock, 3%). Among non-waiting on-CPU time, the top fse.sys frame is Fse!FsePortPoolForEachReservationThatContainsPort, at about 3–5% of the process samples.
The real-world trace shows the same picture. 94% of the test process's samples are in ExpWaitForSpinLock{Exclusive,Shared}AndAcquire, entered through the same FseShimsProcessSteerNblChain → FsePortPoolGetPartitionsForPortUncached path.
Possible explanation (inferred from function names only): every loopback packet delivered by IppLbIndicatePackets is steered by FSE, which looks up the port's partitions. A cache miss falls through to ...Uncached, which takes a port-pool-wide spin lock exclusively at DISPATCH_LEVEL and appears to walk the reservations that contain the port. That walk would explain why the cost grows with the number of ports in use. Connection churn adds and removes reservations, which presumably keeps the cache cold. Because the waiters spin at DISPATCH_LEVEL, thread priority does not help, and other threads on those cores, including UI threads, are starved.
Workarounds: wsl --shutdown before running loopback-heavy workloads, switching to NAT mode, or limiting concurrency.
Diagnostic Logs
No logs attached. The standalone program above reproduces the problem in under a minute, and the call stacks above come from a WPR CPU sampling trace (wpr -start CPU) of that program. If needed, I can collect collect-wsl-logs.ps1 -LogProfile networking or a CPU trace during the repro and share it privately.
- Ngôn ngữ chính
- C++
- Star
- 34.4k
- Fork
- 1.9k
- Merge trung bình
- 1 ngày 23 giờ
- Pull request đã merge (30 ngày)
- 166
Chuẩn bị môi trường
- Không có Dockerfile hay tệp Docker Compose
- Có mẫu pull request
- Đọc hướng dẫn đóng góp
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Issue khác của microsoft/WSL
-
feature wsl2
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 80/100
microsoft/WSL#42306 · 3 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
feature
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 72/100
microsoft/WSL#42114 · 3 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
feature kernel wsl2
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 68/100
microsoft/WSL#41107 · 3 bình luận · 1 reaction ·
Maintainer thường phản hồi trong vòng 1 ngày
-
feature kernel
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
microsoft/WSL#41054 · 9 bình luận · 1 reaction ·
Maintainer thường phản hồi trong vòng 1 ngày
-
feature GPU
Độ khó 1/5 1-3 giờ Mức phù hợp với người mới 70/100
microsoft/WSL#13592 · 4 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
Tất cả issue của microsoft/WSL
Issue tương tự
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
HarperFast/rocksdb-js#1131 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
area/daemon bug good first issue platform/linux priority/p2
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 72/100
ToshiNamura/Deimos#5 ·
-
needs-format
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
Niko1221/Strata#6251 · 1 bình luận ·
Maintainer thường phản hồi trong vòng 5 ngày
-
[Bug]: NP 管理器文件编辑器里删除键无效Đang mởbug
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 65/100
metasequoiaime/msime#11666 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
bug
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 70/100
patrick-hudson/put.gd-desktop#1679 · 1 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày