Kestrel HTTPS throughput on linux-arm64 about 20 % lower with .NET 11.0.0 than with 10.0.15 (daily-build bisect points to #131177)

Start by reproducing the reported HTTPS throughput difference with the one-file Kestrel app on linux-arm64, then inspect runtime commit #131177, which the daily-build bisect implicates. Compare socket event dispatch with DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1, which recovers most of the loss; verify any suspected cause against the reported .NET 10 and .NET 11 throughput measurements. The report identifies no specific source file or test.

Description

A minimal Kestrel app that answers every HTTPS request with the same 1407 bytes serves about 20 % fewer requests per second on linux-arm64 with .NET 11.0.0 than with .NET 10.0.15. I found it while testing the upgrade of a reverse proxy (YARP on Kestrel) and reduced it to the app below. On arm64 I see the same picture in every setup I tried (Ubuntu and Alpine, JIT and Native AOT): the throughput is lower. I could not reproduce it on x64 or over plain HTTP. A bisect over daily builds points to one specific change as the possible cause; details follow below.

Repro, one file, run once with the .NET 10 SDK and once with the .NET 11 SDK:

#:sdk Microsoft.NET.Sdk.Web
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

using var key = ECDsa.Create();
var request = new CertificateRequest("CN=localhost", key, HashAlgorithmName.SHA256);
using var selfSigned = request.CreateSelfSigned(DateTimeOffset.Now.AddDays(-1), DateTimeOffset.Now.AddDays(1));
var certificate = X509CertificateLoader.LoadPkcs12(selfSigned.Export(X509ContentType.Pfx), null);

var builder = WebApplication.CreateSlimBuilder(args);
builder.Logging.ClearProviders();
builder.WebHost.ConfigureKestrel(kestrel => kestrel.ListenAnyIP(8443, listen => listen.UseHttps(certificate)));

var app = builder.Build();
var body = new string('a', 1407);
app.MapGet("/", () => body);
app.Run();

# server on 2 CPUs, load generator on 4 other CPUs (8 vCPU machine)
docker run -d --rm --name server --network host --cpuset-cpus=0-1 -v "$PWD:/w:ro" -w /tmp \
  mcr.microsoft.com/dotnet/sdk:11.0-resolute dotnet run /w/server.cs        # and sdk:10.0-resolute
docker run --rm --network host --cpuset-cpus=4-7 ghcr.io/hatoo/oha:latest -z 15s -c 50 --insecure --no-tui https://127.0.0.1:8443/   # warm-up
docker run --rm --network host --cpuset-cpus=4-7 ghcr.io/hatoo/oha:latest -z 20s -c 50 --insecure --no-tui https://127.0.0.1:8443/

Result, three runs per SDK, requests per second (range in brackets):

	.NET 10.0.15	.NET 11.0.0	change
sitting 1	60 063 [58 958 - 61 102]	48 875 [47 395 - 49 740]	-18.6 %
sitting 2	60 878 [59 350 - 62 383]	48 881 [48 409 - 49 288]	-19.7 %

I could not measure the difference over plain HTTP or on linux-x64, in the setup used here (loopback, 2 server CPUs, 50
connections). That does not prove that it is absent in other setups; I did not test other hardware, more server CPUs or a
real network.

Configuration

• .NET 10.0.15 and 11.0.0 (mcr.microsoft.com/dotnet/{sdk,aspnet}:{10,11}.0-resolute), JIT
• Ubuntu 26.04.1, kernel 7.0.0-1019-azure, OpenSSL 3.5.5 (libssl.so.3 and libcrypto.so.3 are byte-identical in the .NET 10
  and .NET 11 images)
• linux-arm64: Azure Cobalt 100 VM (Neoverse-N2), 8 vCPU. linux-x64 for comparison: Azure VM, AMD EPYC 9V45, 8 vCPU
• oha 1.16.0, 50 connections, HTTP/1.1, keep-alive

Regression?

Yes, between daily builds of 11.0.0-rc.1. Same app, only the runtime changes (Kestrel app with HTTPS, JIT, linux-arm64,
three runs per build, mean requests per second):

Runtime	rps	against .NET 10
.NET 10.0.15	70 123
11.0.0-preview.7	68 643	-2.1 %
11.0.0-rc.1 / rc.2 / rc.3	54 717 / 56 236 / 56 226	-22.0 % / -19.8 % / -19.8 %
11.0.0	56 295	-19.7 %

Daily build (11.0.0-rc.1.<build>)	VMR commit	rps
26406.108, 26407.111, 26409.102		69 724 / 70 031 / 69 773	good
26410.101	64a29867	67 628	good
26410.109	283314d5	54 229	bad
26411.101, 26411.119	f5782bfe, 7cdb2174	54 060 / 54 097	bad
26411.112	19bc97f3	69 290	good (an older commit, built later)

Data

It is not specific to JIT. A Native AOT build of a similar Kestrel app (HTTPS and HTTP listeners, HTTP/1.1 only, same 1407-byte
response, built with the .NET 10 and the .NET 11 SDK, three runs each) goes from 79 995 to 70 886 rps on the same machine
(-11.4 %); the JIT build of that app loses 20.5 %.

Same bisect, p99 latency: .NET 10 1.3 ms, the good daily builds about 3.4 ms, the bad builds and 11.0.0 about 2 ms.

Effect of switches on 11.0.0 (minimal app above, three runs each, change against .NET 10):

	rps	change
.NET 10.0.15	60 878
.NET 11.0.0	48 881	-19.7 %
DOTNET_ThreadPool_SpuriousDispatchNoSpinThreshold=0 (from #135738)	48 665	-20.1 %
DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1	58 335	-4.2 %

It is not specific to glibc or Ubuntu. The same minimal app with the Alpine SDK images (mcr.microsoft.com/dotnet/sdk:{10,11}.0-alpine,
musl 1.2.6, OpenSSL 3.5.9, byte-identical libssl.so.3 in both) on the same machine, three runs each:

	rps	change
.NET 10.0.15	57 311 [55 494 - 58 544]
.NET 11.0.0	46 367 [45 891 - 46 682]	-19.1 %
DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1	55 158 [54 203 - 55 682]	-3.8 %

No effect (Kestrel app, JIT, HTTPS, linux-arm64, all stay between -17 % and -19 %): DOTNET_TieredPGO=0,
DOTNET_JitAsyncInlining=0, DOTNET_SYSTEM_NET_SECURITY_USELEGACYSSLSTREAMHANDSHAKE=1. DOTNET_TieredCompilation=0 halves
the throughput.

In the reverse proxy (Native AOT, 2 CPUs, 50 connections, a cached static response over HTTPS) the same upgrade
costs about 18 % (47 k to 38-39 k rps; CPU per request 41-42 µs to 49-53 µs). perf stat shows cycles per request +23 % and
branch mispredictions per request +50 % to +86 %, spread over the kernel, libcrypto/libssl and managed code, with no single
function standing out.

I could not reproduce it with a server without Kestrel: a Socket accept loop with an SslStream on a NetworkStream shows
-2 % (AOT) and +1 % (JIT) over HTTPS on linux-arm64. That server is much simpler (one loop per connection, no pipes), so
it may not exercise the code that matters.

Analysis

The 9 VMR commits between 64a29867 (good) and 283314d5 (bad) contain 21 dotnet/runtime commits (26b82502..14b601ec) and
18 dotnet/aspnetcore commits (31b9bcde..92f07e29). The only one that touches the socket engine, the thread pool, TLS,
the JIT for async methods or Kestrel is #131177 (queue-subdispatching removal, 2026-08-07). The aspnetcore range has no
Kestrel change. The range also has updates of roslyn, sdk and arcade and #132057, which I did not rule out.

DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1 recovers most of the loss, which fits a change in how socket events are
dispatched to the thread pool. I did not build the runtime with #131177 reverted, so this is a suspicion from the bisect,
not a confirmed cause.

Related: #135738 (thread pool spinning heuristics in rc.2; a different range), #133672 (runtime async code quality in
Kestrel loops; excluded by the switches above).