SqlTransaction.Dispose() throws NullReferenceException from SqlInternalTransaction.Rollback() when it races the background close of a broken connection
Los mantenedores suelen responder en 2 días
Evaluación
Este issue todavía no se ha evaluado.
Descripción
Describe the bug
A command can fail with a fatal error while it is pending asynchronously inside a local transaction. Two examples are a network failure, and a command timeout whose attention the server never acknowledges (an unresponsive server). Disposing the transaction straight afterwards, in a using block or an ORM closing its data context, can then throw:
Exception message: Object reference not set to an instance of an object.
Stack trace:
at Microsoft.Data.SqlClient.SqlInternalTransaction.Rollback()
at Microsoft.Data.SqlClient.SqlInternalTransaction.Dispose(Boolean disposing)
at Microsoft.Data.SqlClient.SqlInternalTransaction.Dispose()
at Microsoft.Data.SqlClient.SqlTransaction.Dispose(Boolean disposing)
Or, when the race goes the other way:
Exception message: Invalid operation. The connection is closed.
Stack trace:
at Microsoft.Data.SqlClient.SqlInternalTransaction.Rollback()
at Microsoft.Data.SqlClient.SqlInternalTransaction.Dispose(Boolean disposing)
...
Either exception replaces the original SqlException (the timeout or the transport error) that the caller was handling.
We hit the NullReferenceException in production after query timeouts against a SQL Server that had stopped responding. The caller was linq2db's DataConnection.CloseAsync() → DbTransaction.DisposeAsync(). In that trace the frames between Rollback() and the caller are inlined.
Cause
On a fatal error the driver faults the pending operation first, and closes the connection afterwards on another thread:
- Network failure:
TdsParser.ThrowExceptionAndWarning(..., asyncClose: true)completes the task with the exception, then runsSqlConnection.Close()fromTask.Factory.StartNew. - Unacknowledged timeout: the
Task.Delay(AttentionTimeoutSeconds * 1000).ContinueWith(...)inTdsParserStateObject.OnTimeoutCorecallssource.TrySetException(...), then_parser.Connection.BreakConnection(), which callsSqlConnection.Close().
So the caller's dispose runs concurrently with that close. SqlInternalTransaction.Dispose(bool) checks _innerConnection != null and calls Rollback(). Rollback() then reads _innerConnection again, first _innerConnection.IsLockedForBulkCopy (!_parser.MARSOn && ...), then ValidateConnectionForExecute, without the parser lock.
The close zombies the transaction (_innerConnection = null) and then disposes the internal connection (_parser = null). If that happens between the check and the dereference, the result is a NullReferenceException. If the connection is broken but the close has not run yet, ValidateConnectionForExecute throws ClosedConnectionError. Both exceptions are raised outside the try whose catch honours _disposing, so they escape Dispose().
The code is the same on main. #2879 and #1121 (NullReferenceException in GetServerTransactionLevel) look like the same family: _innerConnection cleared by a concurrent close while a transaction operation runs.
To reproduce
The program below runs a pending ExecuteScalarAsync inside BeginTransaction() through a TCP proxy. It breaks the connection, either with a reset or by freezing the proxy, then disposes the transaction immediately. The race window is a few instructions wide, so an EventListener on Microsoft.Data.SqlClient.EventSource sleeps on the disposing thread when the SqlInternalTransaction.Rollback | API scope event fires. That event is emitted after the null check in Dispose(bool) and before the dereference. No SqlClient code is modified. In production the same delay comes from thread-pool scheduling.
Results against SQL Server 2016 on localhost, .NET 10, Windows:
| Version | mode | hold | Outcome |
|---|---|---|---|
| 7.0.0 | reset | 0 ms | 9/10 InvalidOperationException: … connection is closed |
| 7.0.0 | reset | 300 ms | 7/10 NullReferenceException in Rollback() |
| 7.1.1 | reset | 0 ms | 4/10 InvalidOperationException: … connection is closed |
| 7.1.1 | reset | 300 ms | 2/10 NullReferenceException in Rollback() |
| 7.0.0 | freeze (timeout) | 0 ms | 2/8 InvalidOperationException: … connection is closed |
| 7.0.0 | freeze (timeout) | 300 ms | 2/8 NullReferenceException in Rollback() |
The remaining runs are ones where the background close finished before Dispose() started, so the dispose skipped the rollback. Both UseCompatibilityAsyncBehaviour / UseCompatibilityProcessSni settings reproduce it.
Repro.csproj
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.SqlClient" Version="7.1.1" />
</ItemGroup>
</Project>
Program.cs
using System.Collections.Concurrent;
using System.Diagnostics.Tracing;
using System.Net;
using System.Net.Sockets;
using Microsoft.Data.SqlClient;
// Repro: SqlTransaction.Dispose() throws NullReferenceException from SqlInternalTransaction.Rollback() (or
// InvalidOperationException "connection is closed") when it races the background close of a connection that
// broke under a pending async command.
//
// The EventListener below sleeps on the disposing thread when SqlClient writes the
// "SqlInternalTransaction.Rollback | API" trace event, which is emitted after Dispose's null check and before
// Rollback's dereference. That widens the race window; no SqlClient code is changed. holdMs=0 shows the race
// without the hold.
//
// Usage: dotnet run -- "<connection string>" [holdMs=300] [iterations=10] [mode=reset|freeze] [pauseMs=0] [compat=false]
// mode reset: TCP reset while the command is pending
// mode freeze: the server stops responding; CommandTimeout (1 s) expires, the attention is never
// acknowledged, and after the 5 s attention timeout SqlClient faults the task, then closes the connection
// pauseMs: random 0..pauseMs delay between the failure and Dispose
// compat: value of both UseCompatibilityAsyncBehaviour and UseCompatibilityProcessSni
string target = args[0];
int holdMs = args.Length > 1 ? int.Parse(args[1]) : 300;
int iterations = args.Length > 2 ? int.Parse(args[2]) : 10;
string mode = args.Length > 3 ? args[3] : "reset";
int pauseMs = args.Length > 4 ? int.Parse(args[4]) : 0;
bool compat = args.Length > 5 && bool.Parse(args[5]);
AppContext.SetSwitch("Switch.Microsoft.Data.SqlClient.UseCompatibilityAsyncBehaviour", compat);
AppContext.SetSwitch("Switch.Microsoft.Data.SqlClient.UseCompatibilityProcessSni", compat);
var random = new Random(1);
var targetBuilder = new SqlConnectionStringBuilder(target);
string[] hostPort = targetBuilder.DataSource.Split(',');
using var proxy = new ResettingProxy(hostPort[0], hostPort.Length > 1 ? int.Parse(hostPort[1]) : 1433);
proxy.Start();
var viaProxy = new SqlConnectionStringBuilder(target) { DataSource = $"127.0.0.1,{proxy.Port}", Pooling = false };
using var hold = new RollbackHold();
var outcomes = new ConcurrentDictionary<string, int>();
string? firstNre = null;
for (int i = 0; i < iterations; i++)
{
var connection = new SqlConnection(viaProxy.ConnectionString);
await connection.OpenAsync();
var transaction = connection.BeginTransaction();
var pending = Task.Run(async () =>
{
using var command = new SqlCommand("WAITFOR DELAY '00:00:30'; SELECT 1", connection, transaction) { CommandTimeout = 1 };
await command.ExecuteScalarAsync();
});
await Task.Delay(200);
if (mode == "freeze")
{
proxy.Frozen = true; // the server stops responding, the socket stays open
}
else
{
proxy.ResetAll(); // the network drops while the async command is pending
}
string operation;
try { await pending; operation = "no error"; }
catch (Exception ex) { operation = ex.GetType().Name + (ex is SqlException se ? $" {se.Number}" : ""); }
if (pauseMs > 0)
{
await Task.Delay(random.Next(0, pauseMs));
}
// Dispose the transaction straight after the failure, as a using block or an ORM does.
RollbackHold.HoldMs = holdMs;
string dispose;
try { transaction.Dispose(); dispose = "ok"; }
catch (Exception ex)
{
dispose = $"{ex.GetType().Name}: {ex.Message}";
if (ex is NullReferenceException) firstNre ??= ex.ToString();
}
finally { RollbackHold.HoldMs = 0; }
connection.Dispose();
proxy.ResetAll();
proxy.Frozen = false;
outcomes.AddOrUpdate($"command: {operation}; transaction.Dispose(): {dispose}", 1, (_, n) => n + 1);
}
var version = typeof(SqlConnection).Assembly
.GetCustomAttributes(typeof(System.Reflection.AssemblyInformationalVersionAttribute), false)
.OfType<System.Reflection.AssemblyInformationalVersionAttribute>().First().InformationalVersion;
Console.WriteLine($"Microsoft.Data.SqlClient {version}; holdMs={holdMs}");
foreach (var (outcome, count) in outcomes) Console.WriteLine($"{count,4} x {outcome}");
if (firstNre != null) Console.WriteLine(firstNre);
internal sealed class RollbackHold : EventListener
{
public static volatile int HoldMs;
protected override void OnEventSourceCreated(EventSource source)
{
if (source.Name == "Microsoft.Data.SqlClient.EventSource")
{
EnableEvents(source, EventLevel.Verbose, EventKeywords.All);
}
}
protected override void OnEventWritten(EventWrittenEventArgs e)
{
if (HoldMs > 0 && e.Payload is { Count: > 0 } p && p[0] is string s
&& s.StartsWith("SqlInternalTransaction.Rollback | API", StringComparison.Ordinal))
{
Thread.Sleep(HoldMs);
}
}
}
// Forwards TCP to the server; ResetAll() aborts every forwarded connection with RST.
internal sealed class ResettingProxy(string host, int port) : IDisposable
{
private readonly TcpListener _listener = new(IPAddress.Loopback, 0);
private readonly ConcurrentDictionary<int, (TcpClient Client, TcpClient Server)> _live = new();
private int _ids;
public volatile bool Frozen;
public int Port => ((IPEndPoint)_listener.LocalEndpoint).Port;
public void Start()
{
_listener.Start();
_ = Task.Run(async () =>
{
while (true)
{
TcpClient client;
try { client = await _listener.AcceptTcpClientAsync(); }
catch (Exception) { return; }
_ = Task.Run(() => Forward(client));
}
});
}
private async Task Forward(TcpClient client)
{
int id = Interlocked.Increment(ref _ids);
var server = new TcpClient();
try
{
await server.ConnectAsync(host, port);
_live[id] = (client, server);
await Task.WhenAny(Pump(client.GetStream(), server.GetStream()), Pump(server.GetStream(), client.GetStream()));
}
catch (Exception) { }
finally
{
_live.TryRemove(id, out _);
client.Close();
server.Close();
}
}
private async Task Pump(NetworkStream from, NetworkStream to)
{
var buffer = new byte[16384];
try
{
int n;
while ((n = await from.ReadAsync(buffer)) > 0)
{
while (Frozen) await Task.Delay(10);
await to.WriteAsync(buffer.AsMemory(0, n));
}
}
catch (Exception) { }
}
public void ResetAll()
{
foreach (var id in _live.Keys)
{
if (_live.TryRemove(id, out var pair))
{
try { pair.Client.Client.LingerState = new LingerOption(true, 0); pair.Client.Close(); } catch (Exception) { }
try { pair.Server.Close(); } catch (Exception) { }
}
}
}
public void Dispose() => _listener.Stop();
}
Expected behavior
SqlTransaction.Dispose() does not throw because the connection broke. Nothing can be rolled back on a broken connection, the server ends the transaction with the session, and _disposing exists so that dispose-time rollback failures are not thrown.
Further technical details
Microsoft.Data.SqlClient version: 7.0.0 (production), 7.1.1 and main (reproduced)
.NET target: .NET 10
SQL Server version: SQL Server 2019 (production), SQL Server 2016 (repro)
Operating system: Windows Server (production), Windows 10 (repro)
Additional context
I have a fix with deterministic regression tests and will open a PR for it.
- Lenguaje dominante
- C#
- Estrellas
- 1k
- Forks
- 352
- Merge medio
- 4 d 8 h
- PR fusionados (30 d)
- 100
Preparar el entorno
Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.
- Sin Dockerfile ni archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de dotnet/SqlClient
-
Sync ExecuteReader on an STA thread deadlocks against a StateChange handler that marshals to the UI thread (OpenAsyncRetry path)Quizá libre de nuevo @mdaigle la tomó hace 48 días y no hay ningún pull request abierto. AbiertoExternal :link:
Dificultad 2/5 1-3 horas Aptitud para principiantes 64/100
dotnet/SqlClient#4534 · 2 comentarios · 1 asignado ·
Los mantenedores suelen responder en 2 días
-
Tests | Additional test casesAbiertoArea\Tests
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
dotnet/SqlClient#4377 · 2 comentarios ·
Los mantenedores suelen responder en 2 días
-
Server error tokens are lost when SQL Server aborts the connection before the driver has posted its read (bare transport error instead of the server's error)Posiblemente ocupada @poizan42 la tomó hace 1 día. Abierto
Dificultad 4/5 3-5 días Aptitud para principiantes 25/100
dotnet/SqlClient#4994 · 1 comentario ·
Los mantenedores suelen responder en 2 días
-
A major performance regression from when you upgraded 5.2.2 to 7.1.1Posiblemente ocupada @cheenamalhotra la tomó hace 1 día. AbiertoPerformance :chart_with_upwards_trend: Regression :boom:
dotnet/SqlClient#4985 · 3 comentarios · 1 asignado ·
Los mantenedores suelen responder en 2 días
-
Suspected: async MARS command can lose its own fatal error, and sync MARS Read waits for CommandTimeout, after another session tears down the connection (managed SNI)Posiblemente ocupada Un pull request vinculado a esta issue está abierto o ya se fusionó. AbiertoTriage Needed :new:
Dificultad 4/5 3-5 días Aptitud para principiantes 15/100
Los mantenedores suelen responder en 2 días
Todos los issues de dotnet/SqlClient
Issues similares
-
t/bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
dotnet/maui#39950 · 1 reacción ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
CommunityToolkit/Datasync#600 · 1 comentario ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
NimblePros/deviq-hugo#121 ·
Los mantenedores suelen responder en 1 día
-
RANDOM_REPLACE does not keep the date format when the month abbreviation has a trailing periodAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
philterd/phileas-dotnet#188 ·
-
Add All ContributorsAbiertoenhancement ready-for-human
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
fboucher/2d6-dungeon-app#288 · 1 comentario ·
Los mantenedores suelen responder en 1 día