Kestrel HTTPS throughput on linux-arm64 about 20 % lower with .NET 11.0.0 than with 10.0.15 (daily-build bisect points to #131177)

Đang mở
  • Có gì không ổn?
#137,385 4 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
45/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
csharp, linux
Lĩnh vực
backend, networking, performance

Hướng nghiên cứu

Start by reproducing the reported HTTPS throughput difference with the one-file Kestrel app on linux-arm64, then inspect runtime commit #131177, which the daily-build bisect implicates. Compare socket event dispatch with DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1, which recovers most of the loss; verify any suspected cause against the reported .NET 10 and .NET 11 throughput measurements. The report identifies no specific source file or test.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

area-System.Net.Http tenet-performance untriaged
Description

A minimal Kestrel app that answers every HTTPS request with the same 1407 bytes serves about 20 % fewer requests per second on linux-arm64 with .NET 11.0.0 than with .NET 10.0.15. I found it while testing the upgrade of a reverse proxy (YARP on Kestrel) and reduced it to the app below. On arm64 I see the same picture in every setup I tried (Ubuntu and Alpine, JIT and Native AOT): the throughput is lower. I could not reproduce it on x64 or over plain HTTP. A bisect over daily builds points to one specific change as the possible cause; details follow below.

Repro, one file, run once with the .NET 10 SDK and once with the .NET 11 SDK:

#:sdk Microsoft.NET.Sdk.Web
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

using var key = ECDsa.Create();
var request = new CertificateRequest("CN=localhost", key, HashAlgorithmName.SHA256);
using var selfSigned = request.CreateSelfSigned(DateTimeOffset.Now.AddDays(-1), DateTimeOffset.Now.AddDays(1));
var certificate = X509CertificateLoader.LoadPkcs12(selfSigned.Export(X509ContentType.Pfx), null);

var builder = WebApplication.CreateSlimBuilder(args);
builder.Logging.ClearProviders();
builder.WebHost.ConfigureKestrel(kestrel => kestrel.ListenAnyIP(8443, listen => listen.UseHttps(certificate)));

var app = builder.Build();
var body = new string('a', 1407);
app.MapGet("/", () => body);
app.Run();

# server on 2 CPUs, load generator on 4 other CPUs (8 vCPU machine)
docker run -d --rm --name server --network host --cpuset-cpus=0-1 -v "$PWD:/w:ro" -w /tmp \
  mcr.microsoft.com/dotnet/sdk:11.0-resolute dotnet run /w/server.cs        # and sdk:10.0-resolute
docker run --rm --network host --cpuset-cpus=4-7 ghcr.io/hatoo/oha:latest -z 15s -c 50 --insecure --no-tui https://127.0.0.1:8443/   # warm-up
docker run --rm --network host --cpuset-cpus=4-7 ghcr.io/hatoo/oha:latest -z 20s -c 50 --insecure --no-tui https://127.0.0.1:8443/

Result, three runs per SDK, requests per second (range in brackets):

.NET 10.0.15 .NET 11.0.0 change
sitting 1 60 063 [58 958 - 61 102] 48 875 [47 395 - 49 740] -18.6 %
sitting 2 60 878 [59 350 - 62 383] 48 881 [48 409 - 49 288] -19.7 %

I could not measure the difference over plain HTTP or on linux-x64, in the setup used here (loopback, 2 server CPUs, 50
connections). That does not prove that it is absent in other setups; I did not test other hardware, more server CPUs or a
real network.

Configuration
  • .NET 10.0.15 and 11.0.0 (mcr.microsoft.com/dotnet/{sdk,aspnet}:{10,11}.0-resolute), JIT
  • Ubuntu 26.04.1, kernel 7.0.0-1019-azure, OpenSSL 3.5.5 (libssl.so.3 and libcrypto.so.3 are byte-identical in the .NET 10
    and .NET 11 images)
  • linux-arm64: Azure Cobalt 100 VM (Neoverse-N2), 8 vCPU. linux-x64 for comparison: Azure VM, AMD EPYC 9V45, 8 vCPU
  • oha 1.16.0, 50 connections, HTTP/1.1, keep-alive
Regression?

Yes, between daily builds of 11.0.0-rc.1. Same app, only the runtime changes (Kestrel app with HTTPS, JIT, linux-arm64,
three runs per build, mean requests per second):

Runtime rps against .NET 10
.NET 10.0.15 70 123
11.0.0-preview.7 68 643 -2.1 %
11.0.0-rc.1 / rc.2 / rc.3 54 717 / 56 236 / 56 226 -22.0 % / -19.8 % / -19.8 %
11.0.0 56 295 -19.7 %
Daily build (11.0.0-rc.1.<build>) VMR commit rps
26406.108, 26407.111, 26409.102 69 724 / 70 031 / 69 773 good
26410.101 64a29867 67 628 good
26410.109 283314d5 54 229 bad
26411.101, 26411.119 f5782bfe, 7cdb2174 54 060 / 54 097 bad
26411.112 19bc97f3 69 290 good (an older commit, built later)
Data

It is not specific to JIT. A Native AOT build of a similar Kestrel app (HTTPS and HTTP listeners, HTTP/1.1 only, same 1407-byte
response, built with the .NET 10 and the .NET 11 SDK, three runs each) goes from 79 995 to 70 886 rps on the same machine
(-11.4 %); the JIT build of that app loses 20.5 %.

Same bisect, p99 latency: .NET 10 1.3 ms, the good daily builds about 3.4 ms, the bad builds and 11.0.0 about 2 ms.

Effect of switches on 11.0.0 (minimal app above, three runs each, change against .NET 10):

rps change
.NET 10.0.15 60 878
.NET 11.0.0 48 881 -19.7 %
DOTNET_ThreadPool_SpuriousDispatchNoSpinThreshold=0 (from #135738) 48 665 -20.1 %
DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1 58 335 -4.2 %

It is not specific to glibc or Ubuntu. The same minimal app with the Alpine SDK images (mcr.microsoft.com/dotnet/sdk:{10,11}.0-alpine,
musl 1.2.6, OpenSSL 3.5.9, byte-identical libssl.so.3 in both) on the same machine, three runs each:

rps change
.NET 10.0.15 57 311 [55 494 - 58 544]
.NET 11.0.0 46 367 [45 891 - 46 682] -19.1 %
DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1 55 158 [54 203 - 55 682] -3.8 %

No effect (Kestrel app, JIT, HTTPS, linux-arm64, all stay between -17 % and -19 %): DOTNET_TieredPGO=0,
DOTNET_JitAsyncInlining=0, DOTNET_SYSTEM_NET_SECURITY_USELEGACYSSLSTREAMHANDSHAKE=1. DOTNET_TieredCompilation=0 halves
the throughput.

In the reverse proxy (Native AOT, 2 CPUs, 50 connections, a cached static response over HTTPS) the same upgrade
costs about 18 % (47 k to 38-39 k rps; CPU per request 41-42 µs to 49-53 µs). perf stat shows cycles per request +23 % and
branch mispredictions per request +50 % to +86 %, spread over the kernel, libcrypto/libssl and managed code, with no single
function standing out.

I could not reproduce it with a server without Kestrel: a Socket accept loop with an SslStream on a NetworkStream shows
-2 % (AOT) and +1 % (JIT) over HTTPS on linux-arm64. That server is much simpler (one loop per connection, no pipes), so
it may not exercise the code that matters.

Analysis

The 9 VMR commits between 64a29867 (good) and 283314d5 (bad) contain 21 dotnet/runtime commits (26b82502..14b601ec) and
18 dotnet/aspnetcore commits (31b9bcde..92f07e29). The only one that touches the socket engine, the thread pool, TLS,
the JIT for async methods or Kestrel is #131177 (queue-subdispatching removal, 2026-08-07). The aspnetcore range has no
Kestrel change. The range also has updates of roslyn, sdk and arcade and #132057, which I did not rule out.

DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS=1 recovers most of the loss, which fits a change in how socket events are
dispatched to the thread pool. I did not build the runtime with #131177 reverted, so this is a suspicion from the bisect,
not a confirmed cause.

Related: #135738 (thread pool spinning heuristics in rc.2; a different range), #133672 (runtime async code quality in
Kestrel loops; excluded by the switches above).

Ngôn ngữ chính
C#
Star
18.4k
Fork
5.8k
Merge trung bình
2 ngày 19 giờ
Pull request đã merge (30 ngày)
590

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của dotnet/runtime

Tất cả issue của dotnet/runtime

Issue tương tự

Thêm issue về C#

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.