Kestrel HTTPS throughput on linux-arm64 about 20 % lower with .NET 11.0.0 than with 10.0.15 (daily-build bisect points to #131177)

Abierto
  • ¿Algo no está bien?
#137,385 4 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
csharp, linux
Área
backend, networking, performance

Línea de trabajo

Start by reproducing the reported HTTPS throughput difference with the one-file Kestrel app on linux-arm64, then inspect runtime commit #131177, which the daily-build bisect implicates. Compare socket event dispatch with DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1, which recovers most of the loss; verify any suspected cause against the reported .NET 10 and .NET 11 throughput measurements. The report identifies no specific source file or test.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area-System.Net.Http tenet-performance untriaged
Description

A minimal Kestrel app that answers every HTTPS request with the same 1407 bytes serves about 20 % fewer requests per second on linux-arm64 with .NET 11.0.0 than with .NET 10.0.15. I found it while testing the upgrade of a reverse proxy (YARP on Kestrel) and reduced it to the app below. On arm64 I see the same picture in every setup I tried (Ubuntu and Alpine, JIT and Native AOT): the throughput is lower. I could not reproduce it on x64 or over plain HTTP. A bisect over daily builds points to one specific change as the possible cause; details follow below.

Repro, one file, run once with the .NET 10 SDK and once with the .NET 11 SDK:

#:sdk Microsoft.NET.Sdk.Web
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

using var key = ECDsa.Create();
var request = new CertificateRequest("CN=localhost", key, HashAlgorithmName.SHA256);
using var selfSigned = request.CreateSelfSigned(DateTimeOffset.Now.AddDays(-1), DateTimeOffset.Now.AddDays(1));
var certificate = X509CertificateLoader.LoadPkcs12(selfSigned.Export(X509ContentType.Pfx), null);

var builder = WebApplication.CreateSlimBuilder(args);
builder.Logging.ClearProviders();
builder.WebHost.ConfigureKestrel(kestrel => kestrel.ListenAnyIP(8443, listen => listen.UseHttps(certificate)));

var app = builder.Build();
var body = new string('a', 1407);
app.MapGet("/", () => body);
app.Run();

# server on 2 CPUs, load generator on 4 other CPUs (8 vCPU machine)
docker run -d --rm --name server --network host --cpuset-cpus=0-1 -v "$PWD:/w:ro" -w /tmp \
  mcr.microsoft.com/dotnet/sdk:11.0-resolute dotnet run /w/server.cs        # and sdk:10.0-resolute
docker run --rm --network host --cpuset-cpus=4-7 ghcr.io/hatoo/oha:latest -z 15s -c 50 --insecure --no-tui https://127.0.0.1:8443/   # warm-up
docker run --rm --network host --cpuset-cpus=4-7 ghcr.io/hatoo/oha:latest -z 20s -c 50 --insecure --no-tui https://127.0.0.1:8443/

Result, three runs per SDK, requests per second (range in brackets):

.NET 10.0.15 .NET 11.0.0 change
sitting 1 60 063 [58 958 - 61 102] 48 875 [47 395 - 49 740] -18.6 %
sitting 2 60 878 [59 350 - 62 383] 48 881 [48 409 - 49 288] -19.7 %

I could not measure the difference over plain HTTP or on linux-x64, in the setup used here (loopback, 2 server CPUs, 50
connections). That does not prove that it is absent in other setups; I did not test other hardware, more server CPUs or a
real network.

Configuration
  • .NET 10.0.15 and 11.0.0 (mcr.microsoft.com/dotnet/{sdk,aspnet}:{10,11}.0-resolute), JIT
  • Ubuntu 26.04.1, kernel 7.0.0-1019-azure, OpenSSL 3.5.5 (libssl.so.3 and libcrypto.so.3 are byte-identical in the .NET 10
    and .NET 11 images)
  • linux-arm64: Azure Cobalt 100 VM (Neoverse-N2), 8 vCPU. linux-x64 for comparison: Azure VM, AMD EPYC 9V45, 8 vCPU
  • oha 1.16.0, 50 connections, HTTP/1.1, keep-alive
Regression?

Yes, between daily builds of 11.0.0-rc.1. Same app, only the runtime changes (Kestrel app with HTTPS, JIT, linux-arm64,
three runs per build, mean requests per second):

Runtime rps against .NET 10
.NET 10.0.15 70 123
11.0.0-preview.7 68 643 -2.1 %
11.0.0-rc.1 / rc.2 / rc.3 54 717 / 56 236 / 56 226 -22.0 % / -19.8 % / -19.8 %
11.0.0 56 295 -19.7 %
Daily build (11.0.0-rc.1.<build>) VMR commit rps
26406.108, 26407.111, 26409.102 69 724 / 70 031 / 69 773 good
26410.101 64a29867 67 628 good
26410.109 283314d5 54 229 bad
26411.101, 26411.119 f5782bfe, 7cdb2174 54 060 / 54 097 bad
26411.112 19bc97f3 69 290 good (an older commit, built later)
Data

It is not specific to JIT. A Native AOT build of a similar Kestrel app (HTTPS and HTTP listeners, HTTP/1.1 only, same 1407-byte
response, built with the .NET 10 and the .NET 11 SDK, three runs each) goes from 79 995 to 70 886 rps on the same machine
(-11.4 %); the JIT build of that app loses 20.5 %.

Same bisect, p99 latency: .NET 10 1.3 ms, the good daily builds about 3.4 ms, the bad builds and 11.0.0 about 2 ms.

Effect of switches on 11.0.0 (minimal app above, three runs each, change against .NET 10):

rps change
.NET 10.0.15 60 878
.NET 11.0.0 48 881 -19.7 %
DOTNET_ThreadPool_SpuriousDispatchNoSpinThreshold=0 (from #135738) 48 665 -20.1 %
DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1 58 335 -4.2 %

It is not specific to glibc or Ubuntu. The same minimal app with the Alpine SDK images (mcr.microsoft.com/dotnet/sdk:{10,11}.0-alpine,
musl 1.2.6, OpenSSL 3.5.9, byte-identical libssl.so.3 in both) on the same machine, three runs each:

rps change
.NET 10.0.15 57 311 [55 494 - 58 544]
.NET 11.0.0 46 367 [45 891 - 46 682] -19.1 %
DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1 55 158 [54 203 - 55 682] -3.8 %

No effect (Kestrel app, JIT, HTTPS, linux-arm64, all stay between -17 % and -19 %): DOTNET_TieredPGO=0,
DOTNET_JitAsyncInlining=0, DOTNET_SYSTEM_NET_SECURITY_USELEGACYSSLSTREAMHANDSHAKE=1. DOTNET_TieredCompilation=0 halves
the throughput.

In the reverse proxy (Native AOT, 2 CPUs, 50 connections, a cached static response over HTTPS) the same upgrade
costs about 18 % (47 k to 38-39 k rps; CPU per request 41-42 µs to 49-53 µs). perf stat shows cycles per request +23 % and
branch mispredictions per request +50 % to +86 %, spread over the kernel, libcrypto/libssl and managed code, with no single
function standing out.

I could not reproduce it with a server without Kestrel: a Socket accept loop with an SslStream on a NetworkStream shows
-2 % (AOT) and +1 % (JIT) over HTTPS on linux-arm64. That server is much simpler (one loop per connection, no pipes), so
it may not exercise the code that matters.

Analysis

The 9 VMR commits between 64a29867 (good) and 283314d5 (bad) contain 21 dotnet/runtime commits (26b82502..14b601ec) and
18 dotnet/aspnetcore commits (31b9bcde..92f07e29). The only one that touches the socket engine, the thread pool, TLS,
the JIT for async methods or Kestrel is #131177 (queue-subdispatching removal, 2026-08-07). The aspnetcore range has no
Kestrel change. The range also has updates of roslyn, sdk and arcade and #132057, which I did not rule out.

DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1 recovers most of the loss, which fits a change in how socket events are
dispatched to the thread pool. I did not build the runtime with #131177 reverted, so this is a suspicion from the bisect,
not a confirmed cause.

Related: #135738 (thread pool spinning heuristics in rc.2; a different range), #133672 (runtime async code quality in
Kestrel loops; excluded by the switches above).

Lenguaje dominante
C#
Estrellas
18.4k
Forks
5.8k
Merge medio
2 d 19 h
PR fusionados (30 d)
590

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dotnet/runtime

Todos los issues de dotnet/runtime

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.