Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Memory stream queue dequeues time out when target silo is unresponsive; pulling agent retries without recovering

Closed
  • Is something wrong?
#5,704 30 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
backend

Research direction

Read PersistentStreamPullingAgent.ReadFromQueue and MemoryAdapterReceiver.GetQueueMessagesAsync in src/Orleans.Streaming/, then investigate the health of silo 10.244.3.152 during the reported timeout window and cross-reference the sibling incidents listed. The issue reports a transient, self-recovered cluster-wide symptom and does not establish a code change; done would require identifying a root cause and an appropriate fix rather than merely raising the timeout.

Written by the indexing model from the issue text.

Description

bug sev:M

What is failing
The Orleans persistent-stream pulling agents on the portal pods cannot read from any of their memory stream queues: every IMemoryStreamQueueGrain.Dequeue request sent to the queue grains on the target silo 10.244.3.152:11111 exceeds the 30-second default response timeout. The retry wrapper (PersistentStreamPullingAgent.ReadFromQueue via AsyncExecutorWithRetries) logs and retries, but the requests keep timing out for the whole window, so stream delivery through all eight memory queues stalls.

Probable cause
Not a defect in the memory stream adapter itself — confidence high that this is downstream of an unresponsive or overloaded silo. Every sample targets the same silo address and the same memorystreamqueue grain placement, while two different pods (and all queue ids 0–7) fail identically and simultaneously; that is the signature of the target silo's grain scheduler being stalled or the process saturated, not of per-queue corruption. This incident is one of a cluster-wide family of System.TimeoutException incidents in the same namespace over 2026-09-22 → 09-24 (Messaging "Failed to address message", RoutingGrain directed-delivery failures, ClientDirectory routing-table publish, ActivityTracking, DataContext hub initialization timeouts — e.g. Admin/_LogIncident/f7d8f9982b20cce5, 254804a0aad8b746, af71d14af5a4d50c), which strengthens the read that one silo or the cluster as a whole was degraded. Root cause of that degradation is not determinable from this evidence alone.

Impact
21 occurrences across both portal pods inside a single ~80-second window, then it stops. Stream delivery to subscribers on those queues was stalled for that window; self-recovered. Given the sibling incidents, treat this as one symptom of the larger timeout storm rather than an isolated defect.

Where to look
Start from Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue and Orleans.Providers.MemoryAdapterReceiver.GetQueueMessagesAsync (Orleans framework source, src/Orleans.Streaming/), then investigate the health of the silo at 10.244.3.152 during the window — grain-scheduler queue depth, GC pauses, CPU/memory — and cross-reference with the sibling timeout incidents above rather than fixing them one by one. If the 30s dequeue timeout proves too tight under load, raising the stream-pulling agent's execution timeout is a workaround, not the fix.

Evidence
Fingerprint 2130b40397e707aa
Category Orleans.Streams.Memory
Severity Error
Exception System.TimeoutException
Namespace memex
Pods memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-56fbdcd48f-sz5st
Occurrences 21
First seen 2026-09-24 06:26:15Z
Last seen 2026-09-24 06:27:38Z
Routing not determined — no configured route matches the category Orleans.Streams.Memory. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-09-24 06:27:38Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-0-0x00000000
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.2.41:11111:149226787 sys.svc.stream.agent/10.244.2.41:11111@149226787+Memory_1_memory-0-0x00000000]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590000000000000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3E90DCEC8F877A6B. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-5-0xA0000005
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.2.41:11111:149226787 sys.svc.stream.agent/10.244.2.41:11111@149226787+Memory_3_memory-5-0xA0000005]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb4887559050000a005000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3E90DCEC8F877A68. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-3-0x60000003
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.9.67:11111:149185698 sys.svc.stream.agent/10.244.9.67:11111@149185698+Memory_3_memory-3-0x60000003]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590300006003000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #6E969F91DB24842B. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-2-0x40000002
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.9.67:11111:149185698 sys.svc.stream.agent/10.244.9.67:11111@149185698+Memory_7_memory-2-0x40000002]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590200004002000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #6E969F91DB24842A. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-4-0x80000004
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.9.67:11111:149185698 sys.svc.stream.agent/10.244.9.67:11111@149185698+Memory_4_memory-4-0x80000004]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590400008004000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #6E969F91DB248437. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241

Re-addressed by the current identity function: this incident inherited 3a8dfa6d2e5c4d2d (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Opened automatically from Admin/_LogIncident/2130b40397e707aa. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 56050d06f832b3ce: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
4h 50m
Merged PRs (30d)
949

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.