Managed SNI: TLS handshake ignores Connect Timeout, one stalled handshake blocks every OpenAsync on the pool

Describe the bug

Hello, with managed SNI the TLS handshake of a new connection is not bounded by Connect Timeout. If the server answers PRELOGIN and then nothing comes back during the TLS handshake, Open() blocks in SslStream.AuthenticateAsClient until the OS gives up on the socket, on Linux that is like 15 minutes by default (I measured 115 s with tcp_retries2=8), and it never ends if the other side keeps acking, which is what the repro below does.

With pooling it is worse, because the pool creates one connection at a time. One stalled handshake blocks every other OpenAsync on that pool (sync Open() still gets the pool timeout), even when the server is reachable again and a non pooled open takes milliseconds.

We did hit this on Azure App Service on Linux against Azure SQL. After a short drop on the database side the API did not open a single connection for 18 minutes, until we restarted it. No login timeout and no pool timeout was thrown, the requests only ended on their own cancellation tokens.

Native SNI on Windows honors the timeout, the same test fails after 5 s there.

No exception is thrown while it hangs. This is the blocked thread (6.1.6, Linux, dotnet-stack):

System.Net.Sockets.NetworkStream.Read(Span<byte>)
Microsoft.Data.SqlClient.ManagedSni.SslOverTdsStream.Read(Span<byte>)
System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions)
Microsoft.Data.SqlClient.ManagedSni.SniTcpHandle.EnableSsl(uint)
Microsoft.Data.SqlClient.ManagedSni.TdsParserStateObjectManaged.EnableSsl(uint&, bool, string)
Microsoft.Data.SqlClient.TdsParser.EnableSsl(uint, SqlConnectionEncryptOption, bool, string)
Microsoft.Data.SqlClient.TdsParser.ConsumePreLoginHandshake(...)
Microsoft.Data.SqlClient.TdsParser.Connect(...)
Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(...)
Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(...)
Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(...)
Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(...)
Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(...)
Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(...)
Microsoft.Data.SqlClient.ConnectionPool.WaitHandleDbConnectionPool.CreateObject(...)
Microsoft.Data.SqlClient.ConnectionPool.WaitHandleDbConnectionPool.TryGetConnection(...)
Microsoft.Data.SqlClient.ConnectionPool.WaitHandleDbConnectionPool.WaitForPendingOpen()

The queued OpenAsync calls are all served by this one thread, so they wait behind it.

To reproduce

Needs any SQL Server the program can reach, CONN is its connection string. The program puts a small TCP proxy in front of it. For a new connection the proxy relays the PRELOGIN request and response and then reads and drops everything, so the TLS ClientHello gets no answer. Connect Timeout is 5.

Step 1 is one Open() without pool. Step 2 starts one pooled OpenAsync into the stall, switches the proxy back to normal relaying, and then opens 5 more connections on the same pool. Each step waits 60 s (WAIT_SEC) before it reports the open as still blocked.

Put the two files into an empty folder and run this from it. SQL Server needs like 30 s to start before the last command.

docker network create repro-net
docker run -d --name repro-sql --network repro-net -e ACCEPT_EULA=Y -e "MSSQL_SA_PASSWORD=Repro!Pass12345" mcr.microsoft.com/mssql/server:2022-latest
docker run --rm --network repro-net -e "CONN=Server=repro-sql,1433;User Id=sa;Password=Repro!Pass12345;TrustServerCertificate=True" -v "$PWD:/src" -w /src mcr.microsoft.com/dotnet/sdk:10.0 dotnet run -c Release

On Windows set MANAGED=1 to use managed networking. Other driver version with -p:SqlClientVersion=7.1.1 after dotnet run -c Release.

repro.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <SqlClientVersion Condition="'$(SqlClientVersion)' == ''">6.1.6</SqlClientVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Data.SqlClient" Version="$(SqlClientVersion)" />
  </ItemGroup>
</Project>

Program.cs

using System.Diagnostics;
using System.Net;
using System.Net.Sockets;
using Microsoft.Data.SqlClient;

// CONN = connection string of any reachable SQL Server, e.g.
// Server=localhost,1433;User Id=sa;Password=...;TrustServerCertificate=True
// MANAGED=1 turns managed networking on (only matters on Windows).
if (Environment.GetEnvironmentVariable("MANAGED") == "1")
    AppContext.SetSwitch("Switch.Microsoft.Data.SqlClient.UseManagedNetworkingOnWindows", true);

var real = new SqlConnectionStringBuilder(Environment.GetEnvironmentVariable("CONN"));
var parts = real.DataSource.Replace("tcp:", "").Split(',');
string host = parts[0];
int port = parts.Length > 1 ? int.Parse(parts[1]) : 1433;

// Proxy in front of the server. While blackhole is on it relays the PRELOGIN
// request and response of a new connection and then reads and drops everything,
// so the TLS ClientHello is accepted by TCP and never answered.
bool blackhole = true;
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
int proxyPort = ((IPEndPoint)listener.LocalEndpoint).Port;
_ = Task.Run(async () =>
{
    while (true)
    {
        var client = await listener.AcceptTcpClientAsync();
        _ = Relay(client, blackhole);
    }
});

async Task<int> ReadTdsPacket(NetworkStream s, byte[] buf)
{
    await s.ReadExactlyAsync(buf.AsMemory(0, 8));
    int len = (buf[2] << 8) | buf[3];
    await s.ReadExactlyAsync(buf.AsMemory(8, len - 8));
    return len;
}

async Task Relay(TcpClient client, bool drop)
{
    try
    {
        using var server = new TcpClient();
        await server.ConnectAsync(host, port);
        var c = client.GetStream();
        var s = server.GetStream();
        if (!drop)
        {
            await Task.WhenAny(c.CopyToAsync(s), s.CopyToAsync(c));
            return;
        }
        var buf = new byte[65536];
        int n = await ReadTdsPacket(c, buf);
        await s.WriteAsync(buf.AsMemory(0, n));
        n = await ReadTdsPacket(s, buf);
        await c.WriteAsync(buf.AsMemory(0, n));
        while (await c.ReadAsync(buf) > 0) { }
    }
    catch { }
    finally { client.Dispose(); }
}

string Conn(bool pooling) => new SqlConnectionStringBuilder(real.ConnectionString)
{
    DataSource = $"127.0.0.1,{proxyPort}",
    ConnectTimeout = 5,
    Pooling = pooling,
    Encrypt = true,
}.ConnectionString;

string Describe(Task t) =>
    t.IsCompletedSuccessfully ? "opened" : $"{t.Exception!.InnerException!.GetType().Name}: {t.Exception.InnerException.Message.Split('\n')[0]}";

Console.WriteLine($"SqlClient {System.Diagnostics.FileVersionInfo.GetVersionInfo(typeof(SqlConnection).Assembly.Location).ProductVersion}, {System.Runtime.InteropServices.RuntimeInformation.OSDescription}, Connect Timeout=5");

// 1. One open, no pool. The server answers PRELOGIN, the TLS handshake gets no answer.
var sw = Stopwatch.StartNew();
var single = Task.Run(() => { using var c = new SqlConnection(Conn(false)); c.Open(); });
int waitSec = int.Parse(Environment.GetEnvironmentVariable("WAIT_SEC") ?? "60");
if (await Task.WhenAny(single, Task.Delay(TimeSpan.FromSeconds(waitSec))) == single)
    Console.WriteLine($"1. Open() ended after {sw.Elapsed.TotalSeconds:F1}s: {Describe(single)}");
else
    Console.WriteLine($"1. Open() still blocked after {sw.Elapsed.TotalSeconds:F1}s");

// 2. Pool. The first open hits the same stall, then the proxy relays normally again.
var stuck = Task.Run(async () => { await using var c = new SqlConnection(Conn(true)); await c.OpenAsync(); });
await Task.Delay(2000);
blackhole = false;

sw.Restart();
await using (var direct = new SqlConnection(Conn(false)))
{
    await direct.OpenAsync();
    Console.WriteLine($"2. proxy relays again, open without pool took {sw.ElapsedMilliseconds}ms");
}

var pooled = Enumerable.Range(0, 5).Select(i => Task.Run(async () =>
{
    var one = Stopwatch.StartNew();
    try
    {
        await using var c = new SqlConnection(Conn(true));
        await c.OpenAsync();
        return $"opened after {one.Elapsed.TotalSeconds:F1}s";
    }
    catch (Exception ex)
    {
        return $"{ex.GetType().Name} after {one.Elapsed.TotalSeconds:F1}s: {ex.Message.Split('\n')[0]}";
    }
})).ToArray();

sw.Restart();
if (await Task.WhenAny(Task.WhenAll(pooled), Task.Delay(TimeSpan.FromSeconds(waitSec))) is not Task<string[]>)
    Console.WriteLine($"2. {pooled.Count(p => !p.IsCompleted)} of 5 pooled OpenAsync still waiting after {sw.Elapsed.TotalSeconds:F1}s, first one {(stuck.IsCompleted ? "ended" : "still blocked")}");
foreach (var p in pooled.Where(p => p.IsCompleted)) Console.WriteLine($"2. pooled OpenAsync {p.Result}");
Environment.Exit(0);

Output on Linux (6.1.6, same with 6.1.8, 7.1.1 and 8.0.0-preview1.26313.4, and on Windows with MANAGED=1):

SqlClient 6.1.6+b862260e80dd3d159b812cbe5c51b933fe348afe, Ubuntu 24.04.5 LTS, Connect Timeout=5
1. Open() still blocked after 60.0s
2. proxy relays again, open without pool took 64ms
2. 5 of 5 pooled OpenAsync still waiting after 60.0s, first one still blocked

Output on Windows with native SNI:

SqlClient 6.1.6+b862260e80dd3d159b812cbe5c51b933fe348afe, Microsoft Windows 10.0.26300, Connect Timeout=5
1. Open() ended after 5.0s: SqlException: A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 0 - The wait operation timed out.)
2. proxy relays again, open without pool took 2071ms
2. pooled OpenAsync SqlException after 0.9s: A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 0 - The wait operation timed out.)
(same line for all 5)

The 5 pooled opens fail there with the error of the first one, I guess that is the pool handing out the cached creation error.

Expected behavior

Open() / OpenAsync() fails when Connect Timeout elapses, also when the time is spent in the TLS handshake, same as native SNI does. The PRELOGIN read before it does honor the timeout, when PRELOGIN gets no answer the open fails with Connection Timeout Expired as expected.

And one stuck connection attempt should not keep the other opens on the pool waiting past their own timeout.

Further technical details

Microsoft.Data.SqlClient version: 6.1.6, also 6.1.8, 7.1.1 and 8.0.0-preview1.26313.4
.NET target: .NET 10
SQL Server version: SQL Server 2022 in Docker for the repro, Azure SQL Database where we hit it
Operating system: Linux (mcr.microsoft.com/dotnet/sdk:10.0, Ubuntu 24.04), Azure App Service on Linux, Windows 11 with managed networking

Additional context
SniTcpHandle.EnableSsl calls _sslStream.AuthenticateAsClient(...) with no timeout or cancellation, and I did not find the socket receive timeout being set for that step. Possibly related to #979, since OpenAsync runs this synchronously.

Is there any workaround other than lowering the TCP timeouts of the OS? Thanks.