SignalR: keep-alive ping and stateful reconnect ack race on Transport.Output, corrupting the stream

Start with HubConnectionContext.cs and MessageBuffer.cs on the release/10.0 branch, tracing the ping writers and MessageBuffer.RunTimer to understand how they write to Transport.Output. Then inspect the related tests and reproduce the overlap between a keep-alive ping and a stateful reconnect acknowledgment. Done means concurrent writes cannot corrupt the stream and a regression test covers the race.

Is there an existing issue for this?

I searched the issues and found nothing matching. PR #65732 touches MessageBuffer write cancellation but not this race.

Describe the bug

With stateful reconnect on, the server can write a keep-alive ping and a stateful reconnect AckMessage to the same PipeWriter at the same time. Each writer holds a different lock, so nothing serializes them. The bytes mix on the wire, and the client fails to parse the stream and drops the connection.

The two writers in release/10.0:

Writer | Lock held | Writes to -- | -- | -- HubConnectionContext.TryWritePingAsync / TryWritePingSlowAsync (HubConnectionContext.cs) | HubConnectionContext._writeLock | _connectionContext.Transport.Output MessageBuffer.RunTimer (MessageBuffer.cs), every AckRate (1 s) when _lastAckedId < _latestReceivedSequenceId | MessageBuffer._writeLock | _writer, set from connection.Transport.Output in the constructor

Normal hub messages are safe: HubConnectionContext.WriteCore calls _messageBuffer.WriteAsync while holding HubConnectionContext._writeLock, and MessageBuffer then takes its own lock. But the ping path never takes MessageBuffer._writeLock, and the ack timer never takes HubConnectionContext._writeLock. The ping path doesn't check UsingStatefulReconnect() either, so it bypasses the buffer.

The window opens when the server has sent nothing for KeepAliveInterval, so a ping is due, while the client keeps sending messages that need an ack, for example the reply to a long-running server-to-client request.

Expected behavior

Pings and acks are serialized with each other and with normal hub messages, so the stream stays well formed.

Steps to reproduce

We have no minimal repro yet. Observed setup:

• ASP.NET Core server hub with AllowStatefulReconnects = true, KeepAliveInterval = 8 s, ClientTimeoutInterval = 17.6 s, JSON protocol.
• .NET client (Microsoft.AspNetCore.SignalR.Client 10.0.13) with .WithStatefulReconnect(), .WithAutomaticReconnect(...), WithKeepAliveInterval(8 s), WithServerTimeout(17.6 s), WebSockets transport over localhost.
• The server calls client methods through IHubContext<T>.Clients.All.SendCoreAsync, and the client replies by invoking a hub method.
• The connection drops 10 to 15 times a day at irregular times. We are shipping a build with stateful reconnect off and will report whether the drops stop.

A repro would likely need a server that sends nothing while the client invokes hub methods often, with a short KeepAliveInterval to make collisions more frequent.

Exceptions (if any)

Client side, raised as HubConnection.Reconnecting. The first character changes from drop to drop: 's', 'o', and an empty message.

System.IO.InvalidDataException: Error reading JSON.
 ---> System.Text.Json.JsonReaderException: 's' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
   at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
   at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
   at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first)
   at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
   at Microsoft.AspNetCore.SignalR.Protocol.JsonHubProtocol.ParseMessage(ReadOnlySequence`1 input, IInvocationBinder binder)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.SignalR.Protocol.JsonHubProtocol.ParseMessage(ReadOnlySequence`1 input, IInvocationBinder binder)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReceiveLoop(ConnectionState connectionState)

System.IO.InvalidDataException: Error reading JSON.
 ---> System.Text.Json.JsonReaderException: The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. LineNumber: 0 | BytePositionInLine: 0.
   at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
   at Microsoft.AspNetCore.SignalR.Protocol.JsonHubProtocol.ParseMessage(ReadOnlySequence`1 input, IInvocationBinder binder)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.SignalR.Protocol.JsonHubProtocol.ParseMessage(ReadOnlySequence`1 input, IInvocationBinder binder)
   at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReceiveLoop(ConnectionState connectionState)

The server logs only a normal disconnect, with no exception.

.NET Version

• Server: Microsoft.AspNetCore.App 10.0.14, Windows, Kestrel.
• Client: Microsoft.AspNetCore.SignalR.Client 10.0.13, net10.0.

Anything else?

Possible fixes: have the ack timer take HubConnectionContext._writeLock, for example by passing that lock into MessageBuffer. Or, when stateful reconnect is on, route pings through MessageBuffer under its lock.

We haven't checked whether the .NET client has the same ping and ack pattern in the other direction.

Workaround: turn off stateful reconnect.

Is there an existing issue for this? I searched the issues and found nothing matching. PR #65732 touches MessageBuffer write cancellation but not this race.

Describe the bug
With stateful reconnect on, the server can write a keep-alive ping and a stateful reconnect AckMessage to the same PipeWriter at the same time. Each writer holds a different lock, so nothing serializes them. The bytes mix on the wire, and the client fails to parse the stream and drops the connection.

The two writers in release/10.0:

Writer Lock held Writes to
HubConnectionContext.TryWritePingAsync / TryWritePingSlowAsync (HubConnectionContext.cs) HubConnectionContext._writeLock _connectionContext.Transport.Output
MessageBuffer.RunTimer (MessageBuffer.cs), every AckRate (1 s) when _lastAckedId < _latestReceivedSequenceId MessageBuffer._writeLock _writer, set from connection.Transport.Output in the constructor
Normal hub messages are safe: HubConnectionContext.WriteCore calls _messageBuffer.WriteAsync while holding HubConnectionContext._writeLock, and MessageBuffer then takes its own lock. But the ping path never takes MessageBuffer._writeLock, and the ack timer never takes HubConnectionContext._writeLock. The ping path doesn't check UsingStatefulReconnect() either, so it bypasses the buffer.

The window opens when the server has sent nothing for KeepAliveInterval, so a ping is due, while the client keeps sending messages that need an ack, for example the reply to a long-running server-to-client request.

Expected behavior
Pings and acks are serialized with each other and with normal hub messages, so the stream stays well formed.

Steps to reproduce
We have no minimal repro yet. Observed setup:

ASP.NET Core server hub with AllowStatefulReconnects = true, KeepAliveInterval = 8 s, ClientTimeoutInterval = 17.6 s, JSON protocol.
.NET client (Microsoft.AspNetCore.SignalR.Client 10.0.13) with .WithStatefulReconnect(), .WithAutomaticReconnect(...), WithKeepAliveInterval(8 s), WithServerTimeout(17.6 s), WebSockets transport over localhost.
The server calls client methods through IHubContext.Clients.All.SendCoreAsync, and the client replies by invoking a hub method.
The connection drops 10 to 15 times a day at irregular times. We are shipping a build with stateful reconnect off and will report whether the drops stop.
A repro would likely need a server that sends nothing while the client invokes hub methods often, with a short KeepAliveInterval to make collisions more frequent.

Exceptions (if any)
Client side, raised as HubConnection.Reconnecting. The first character changes from drop to drop: 's', 'o', and an empty message.

System.IO.InvalidDataException: Error reading JSON.
---> System.Text.Json.JsonReaderException: 's' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan1 bytes) at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker) at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first) at System.Text.Json.Utf8JsonReader.ReadSingleSegment() at Microsoft.AspNetCore.SignalR.Protocol.JsonHubProtocol.ParseMessage(ReadOnlySequence1 input, IInvocationBinder binder)
--- End of inner exception stack trace ---
at Microsoft.AspNetCore.SignalR.Protocol.JsonHubProtocol.ParseMessage(ReadOnlySequence1 input, IInvocationBinder binder) at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReceiveLoop(ConnectionState connectionState) System.IO.InvalidDataException: Error reading JSON. ---> System.Text.Json.JsonReaderException: The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. LineNumber: 0 | BytePositionInLine: 0. at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan1 bytes)
at Microsoft.AspNetCore.SignalR.Protocol.JsonHubProtocol.ParseMessage(ReadOnlySequence1 input, IInvocationBinder binder) --- End of inner exception stack trace --- at Microsoft.AspNetCore.SignalR.Protocol.JsonHubProtocol.ParseMessage(ReadOnlySequence1 input, IInvocationBinder binder)
at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReceiveLoop(ConnectionState connectionState)
The server logs only a normal disconnect, with no exception.

.NET Version
Server: Microsoft.AspNetCore.App 10.0.14, Windows, Kestrel.
Client: Microsoft.AspNetCore.SignalR.Client 10.0.13, net10.0.
Anything else?
Possible fixes: have the ack timer take HubConnectionContext._writeLock, for example by passing that lock into MessageBuffer. Or, when stateful reconnect is on, route pings through MessageBuffer under its lock.

We haven't checked whether the .NET client has the same ping and ack pattern in the other direction.

Workaround: turn off stateful reconnect.