SqlTransaction.Dispose() throws NullReferenceException from SqlInternalTransaction.Rollback() when it races the background close of a broken connection
I maintainer di solito rispondono entro 2 giorni
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
Describe the bug
A command can fail with a fatal error while it is pending asynchronously inside a local transaction. Two examples are a network failure, and a command timeout whose attention the server never acknowledges (an unresponsive server). Disposing the transaction straight afterwards, in a using block or an ORM closing its data context, can then throw:
Exception message: Object reference not set to an instance of an object.
Stack trace:
at Microsoft.Data.SqlClient.SqlInternalTransaction.Rollback()
at Microsoft.Data.SqlClient.SqlInternalTransaction.Dispose(Boolean disposing)
at Microsoft.Data.SqlClient.SqlInternalTransaction.Dispose()
at Microsoft.Data.SqlClient.SqlTransaction.Dispose(Boolean disposing)
Or, when the race goes the other way:
Exception message: Invalid operation. The connection is closed.
Stack trace:
at Microsoft.Data.SqlClient.SqlInternalTransaction.Rollback()
at Microsoft.Data.SqlClient.SqlInternalTransaction.Dispose(Boolean disposing)
...
Either exception replaces the original SqlException (the timeout or the transport error) that the caller was handling.
We hit the NullReferenceException in production after query timeouts against a SQL Server that had stopped responding. The caller was linq2db's DataConnection.CloseAsync() → DbTransaction.DisposeAsync(). In that trace the frames between Rollback() and the caller are inlined.
Cause
On a fatal error the driver faults the pending operation first, and closes the connection afterwards on another thread:
- Network failure:
TdsParser.ThrowExceptionAndWarning(..., asyncClose: true)completes the task with the exception, then runsSqlConnection.Close()fromTask.Factory.StartNew. - Unacknowledged timeout: the
Task.Delay(AttentionTimeoutSeconds * 1000).ContinueWith(...)inTdsParserStateObject.OnTimeoutCorecallssource.TrySetException(...), then_parser.Connection.BreakConnection(), which callsSqlConnection.Close().
So the caller's dispose runs concurrently with that close. SqlInternalTransaction.Dispose(bool) checks _innerConnection != null and calls Rollback(). Rollback() then reads _innerConnection again, first _innerConnection.IsLockedForBulkCopy (!_parser.MARSOn && ...), then ValidateConnectionForExecute, without the parser lock.
The close zombies the transaction (_innerConnection = null) and then disposes the internal connection (_parser = null). If that happens between the check and the dereference, the result is a NullReferenceException. If the connection is broken but the close has not run yet, ValidateConnectionForExecute throws ClosedConnectionError. Both exceptions are raised outside the try whose catch honours _disposing, so they escape Dispose().
The code is the same on main. #2879 and #1121 (NullReferenceException in GetServerTransactionLevel) look like the same family: _innerConnection cleared by a concurrent close while a transaction operation runs.
To reproduce
The program below runs a pending ExecuteScalarAsync inside BeginTransaction() through a TCP proxy. It breaks the connection, either with a reset or by freezing the proxy, then disposes the transaction immediately. The race window is a few instructions wide, so an EventListener on Microsoft.Data.SqlClient.EventSource sleeps on the disposing thread when the SqlInternalTransaction.Rollback | API scope event fires. That event is emitted after the null check in Dispose(bool) and before the dereference. No SqlClient code is modified. In production the same delay comes from thread-pool scheduling.
Results against SQL Server 2016 on localhost, .NET 10, Windows:
| Version | mode | hold | Outcome |
|---|---|---|---|
| 7.0.0 | reset | 0 ms | 9/10 InvalidOperationException: … connection is closed |
| 7.0.0 | reset | 300 ms | 7/10 NullReferenceException in Rollback() |
| 7.1.1 | reset | 0 ms | 4/10 InvalidOperationException: … connection is closed |
| 7.1.1 | reset | 300 ms | 2/10 NullReferenceException in Rollback() |
| 7.0.0 | freeze (timeout) | 0 ms | 2/8 InvalidOperationException: … connection is closed |
| 7.0.0 | freeze (timeout) | 300 ms | 2/8 NullReferenceException in Rollback() |
The remaining runs are ones where the background close finished before Dispose() started, so the dispose skipped the rollback. Both UseCompatibilityAsyncBehaviour / UseCompatibilityProcessSni settings reproduce it.
Repro.csproj
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.SqlClient" Version="7.1.1" />
</ItemGroup>
</Project>
Program.cs
using System.Collections.Concurrent;
using System.Diagnostics.Tracing;
using System.Net;
using System.Net.Sockets;
using Microsoft.Data.SqlClient;
// Repro: SqlTransaction.Dispose() throws NullReferenceException from SqlInternalTransaction.Rollback() (or
// InvalidOperationException "connection is closed") when it races the background close of a connection that
// broke under a pending async command.
//
// The EventListener below sleeps on the disposing thread when SqlClient writes the
// "SqlInternalTransaction.Rollback | API" trace event, which is emitted after Dispose's null check and before
// Rollback's dereference. That widens the race window; no SqlClient code is changed. holdMs=0 shows the race
// without the hold.
//
// Usage: dotnet run -- "<connection string>" [holdMs=300] [iterations=10] [mode=reset|freeze] [pauseMs=0] [compat=false]
// mode reset: TCP reset while the command is pending
// mode freeze: the server stops responding; CommandTimeout (1 s) expires, the attention is never
// acknowledged, and after the 5 s attention timeout SqlClient faults the task, then closes the connection
// pauseMs: random 0..pauseMs delay between the failure and Dispose
// compat: value of both UseCompatibilityAsyncBehaviour and UseCompatibilityProcessSni
string target = args[0];
int holdMs = args.Length > 1 ? int.Parse(args[1]) : 300;
int iterations = args.Length > 2 ? int.Parse(args[2]) : 10;
string mode = args.Length > 3 ? args[3] : "reset";
int pauseMs = args.Length > 4 ? int.Parse(args[4]) : 0;
bool compat = args.Length > 5 && bool.Parse(args[5]);
AppContext.SetSwitch("Switch.Microsoft.Data.SqlClient.UseCompatibilityAsyncBehaviour", compat);
AppContext.SetSwitch("Switch.Microsoft.Data.SqlClient.UseCompatibilityProcessSni", compat);
var random = new Random(1);
var targetBuilder = new SqlConnectionStringBuilder(target);
string[] hostPort = targetBuilder.DataSource.Split(',');
using var proxy = new ResettingProxy(hostPort[0], hostPort.Length > 1 ? int.Parse(hostPort[1]) : 1433);
proxy.Start();
var viaProxy = new SqlConnectionStringBuilder(target) { DataSource = $"127.0.0.1,{proxy.Port}", Pooling = false };
using var hold = new RollbackHold();
var outcomes = new ConcurrentDictionary<string, int>();
string? firstNre = null;
for (int i = 0; i < iterations; i++)
{
var connection = new SqlConnection(viaProxy.ConnectionString);
await connection.OpenAsync();
var transaction = connection.BeginTransaction();
var pending = Task.Run(async () =>
{
using var command = new SqlCommand("WAITFOR DELAY '00:00:30'; SELECT 1", connection, transaction) { CommandTimeout = 1 };
await command.ExecuteScalarAsync();
});
await Task.Delay(200);
if (mode == "freeze")
{
proxy.Frozen = true; // the server stops responding, the socket stays open
}
else
{
proxy.ResetAll(); // the network drops while the async command is pending
}
string operation;
try { await pending; operation = "no error"; }
catch (Exception ex) { operation = ex.GetType().Name + (ex is SqlException se ? $" {se.Number}" : ""); }
if (pauseMs > 0)
{
await Task.Delay(random.Next(0, pauseMs));
}
// Dispose the transaction straight after the failure, as a using block or an ORM does.
RollbackHold.HoldMs = holdMs;
string dispose;
try { transaction.Dispose(); dispose = "ok"; }
catch (Exception ex)
{
dispose = $"{ex.GetType().Name}: {ex.Message}";
if (ex is NullReferenceException) firstNre ??= ex.ToString();
}
finally { RollbackHold.HoldMs = 0; }
connection.Dispose();
proxy.ResetAll();
proxy.Frozen = false;
outcomes.AddOrUpdate($"command: {operation}; transaction.Dispose(): {dispose}", 1, (_, n) => n + 1);
}
var version = typeof(SqlConnection).Assembly
.GetCustomAttributes(typeof(System.Reflection.AssemblyInformationalVersionAttribute), false)
.OfType<System.Reflection.AssemblyInformationalVersionAttribute>().First().InformationalVersion;
Console.WriteLine($"Microsoft.Data.SqlClient {version}; holdMs={holdMs}");
foreach (var (outcome, count) in outcomes) Console.WriteLine($"{count,4} x {outcome}");
if (firstNre != null) Console.WriteLine(firstNre);
internal sealed class RollbackHold : EventListener
{
public static volatile int HoldMs;
protected override void OnEventSourceCreated(EventSource source)
{
if (source.Name == "Microsoft.Data.SqlClient.EventSource")
{
EnableEvents(source, EventLevel.Verbose, EventKeywords.All);
}
}
protected override void OnEventWritten(EventWrittenEventArgs e)
{
if (HoldMs > 0 && e.Payload is { Count: > 0 } p && p[0] is string s
&& s.StartsWith("SqlInternalTransaction.Rollback | API", StringComparison.Ordinal))
{
Thread.Sleep(HoldMs);
}
}
}
// Forwards TCP to the server; ResetAll() aborts every forwarded connection with RST.
internal sealed class ResettingProxy(string host, int port) : IDisposable
{
private readonly TcpListener _listener = new(IPAddress.Loopback, 0);
private readonly ConcurrentDictionary<int, (TcpClient Client, TcpClient Server)> _live = new();
private int _ids;
public volatile bool Frozen;
public int Port => ((IPEndPoint)_listener.LocalEndpoint).Port;
public void Start()
{
_listener.Start();
_ = Task.Run(async () =>
{
while (true)
{
TcpClient client;
try { client = await _listener.AcceptTcpClientAsync(); }
catch (Exception) { return; }
_ = Task.Run(() => Forward(client));
}
});
}
private async Task Forward(TcpClient client)
{
int id = Interlocked.Increment(ref _ids);
var server = new TcpClient();
try
{
await server.ConnectAsync(host, port);
_live[id] = (client, server);
await Task.WhenAny(Pump(client.GetStream(), server.GetStream()), Pump(server.GetStream(), client.GetStream()));
}
catch (Exception) { }
finally
{
_live.TryRemove(id, out _);
client.Close();
server.Close();
}
}
private async Task Pump(NetworkStream from, NetworkStream to)
{
var buffer = new byte[16384];
try
{
int n;
while ((n = await from.ReadAsync(buffer)) > 0)
{
while (Frozen) await Task.Delay(10);
await to.WriteAsync(buffer.AsMemory(0, n));
}
}
catch (Exception) { }
}
public void ResetAll()
{
foreach (var id in _live.Keys)
{
if (_live.TryRemove(id, out var pair))
{
try { pair.Client.Client.LingerState = new LingerOption(true, 0); pair.Client.Close(); } catch (Exception) { }
try { pair.Server.Close(); } catch (Exception) { }
}
}
}
public void Dispose() => _listener.Stop();
}
Expected behavior
SqlTransaction.Dispose() does not throw because the connection broke. Nothing can be rolled back on a broken connection, the server ends the transaction with the session, and _disposing exists so that dispose-time rollback failures are not thrown.
Further technical details
Microsoft.Data.SqlClient version: 7.0.0 (production), 7.1.1 and main (reproduced)
.NET target: .NET 10
SQL Server version: SQL Server 2019 (production), SQL Server 2016 (repro)
Operating system: Windows Server (production), Windows 10 (repro)
Additional context
I have a fix with deterministic regression tests and will open a PR for it.
- Lingua principale
- C#
- Stelle
- 1k
- Fork
- 352
- Merge medio
- 4g 8h
- PR unite (30g)
- 104
Preparare l'ambiente
Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di dotnet/SqlClient
-
Sync ExecuteReader on an STA thread deadlocks against a StateChange handler that marshals to the UI thread (OpenAsyncRetry path)Forse di nuovo libera @mdaigle l’ha presa 49 giorni fa e non c’è nessuna pull request aperta. ApertaExternal :link:
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
dotnet/SqlClient#4534 · 2 commenti · 1 assegnatario ·
I maintainer di solito rispondono entro 2 giorni
-
Area\Tests
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
dotnet/SqlClient#4377 · 2 commenti ·
I maintainer di solito rispondono entro 2 giorni
-
Server error tokens are lost when SQL Server aborts the connection before the driver has posted its read (bare transport error instead of the server's error)Forse già presa @poizan42 l’ha presa 2 giorni fa. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 25/100
dotnet/SqlClient#4994 · 1 commento ·
I maintainer di solito rispondono entro 2 giorni
-
A major performance regression from when you upgraded 5.2.2 to 7.1.1Forse già presa @cheenamalhotra l’ha presa 2 giorni fa. ApertaPerformance :chart_with_upwards_trend: Regression :boom:
dotnet/SqlClient#4985 · 3 commenti · 1 assegnatario ·
I maintainer di solito rispondono entro 2 giorni
-
Suspected: async MARS command can lose its own fatal error, and sync MARS Read waits for CommandTimeout, after another session tears down the connection (managed SNI)Forse già presa Una pull request collegata a questa issue è aperta o già unita. ApertaTriage Needed :new:
Difficoltà 4/5 3-5 giorni Idoneità per principianti 15/100
I maintainer di solito rispondono entro 2 giorni
Tutte le issue di dotnet/SqlClient
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
NethermindEth/nethermind#15684 ·
I maintainer di solito rispondono entro 1 giorno
-
area/ai-assistant squad/flyt
Difficoltà 2/5 1-3 ore Idoneità per principianti 83/100
Altinn/altinn-studio#21798 ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
arastella625/DAORework#181 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 77/100
BEagle1984/silverback#280 ·
-
bug P1
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
endercodezz/BYDLO#417 ·
I maintainer di solito rispondono entro 1 giorno