Memory stream queue dequeues time out when target silo is unresponsive; pulling agent retries without recovering

Chiusa
  • Qualcosa non va?
#5,704 30 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
backend

Direzione di ricerca

Read PersistentStreamPullingAgent.ReadFromQueue and MemoryAdapterReceiver.GetQueueMessagesAsync in src/Orleans.Streaming/, then investigate the health of silo 10.244.3.152 during the reported timeout window and cross-reference the sibling incidents listed. The issue reports a transient, self-recovered cluster-wide symptom and does not establish a code change; done would require identifying a root cause and an appropriate fix rather than merely raising the timeout.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:M

What is failing
The Orleans persistent-stream pulling agents on the portal pods cannot read from any of their memory stream queues: every IMemoryStreamQueueGrain.Dequeue request sent to the queue grains on the target silo 10.244.3.152:11111 exceeds the 30-second default response timeout. The retry wrapper (PersistentStreamPullingAgent.ReadFromQueue via AsyncExecutorWithRetries) logs and retries, but the requests keep timing out for the whole window, so stream delivery through all eight memory queues stalls.

Probable cause
Not a defect in the memory stream adapter itself — confidence high that this is downstream of an unresponsive or overloaded silo. Every sample targets the same silo address and the same memorystreamqueue grain placement, while two different pods (and all queue ids 0–7) fail identically and simultaneously; that is the signature of the target silo's grain scheduler being stalled or the process saturated, not of per-queue corruption. This incident is one of a cluster-wide family of System.TimeoutException incidents in the same namespace over 2026-09-22 → 09-24 (Messaging "Failed to address message", RoutingGrain directed-delivery failures, ClientDirectory routing-table publish, ActivityTracking, DataContext hub initialization timeouts — e.g. Admin/_LogIncident/f7d8f9982b20cce5, 254804a0aad8b746, af71d14af5a4d50c), which strengthens the read that one silo or the cluster as a whole was degraded. Root cause of that degradation is not determinable from this evidence alone.

Impact
21 occurrences across both portal pods inside a single ~80-second window, then it stops. Stream delivery to subscribers on those queues was stalled for that window; self-recovered. Given the sibling incidents, treat this as one symptom of the larger timeout storm rather than an isolated defect.

Where to look
Start from Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue and Orleans.Providers.MemoryAdapterReceiver.GetQueueMessagesAsync (Orleans framework source, src/Orleans.Streaming/), then investigate the health of the silo at 10.244.3.152 during the window — grain-scheduler queue depth, GC pauses, CPU/memory — and cross-reference with the sibling timeout incidents above rather than fixing them one by one. If the 30s dequeue timeout proves too tight under load, raising the stream-pulling agent's execution timeout is a workaround, not the fix.

Evidence
Fingerprint 2130b40397e707aa
Category Orleans.Streams.Memory
Severity Error
Exception System.TimeoutException
Namespace memex
Pods memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-56fbdcd48f-sz5st
Occurrences 21
First seen 2026-09-24 06:26:15Z
Last seen 2026-09-24 06:27:38Z
Routing not determined — no configured route matches the category Orleans.Streams.Memory. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-09-24 06:27:38Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-0-0x00000000
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.2.41:11111:149226787 sys.svc.stream.agent/10.244.2.41:11111@149226787+Memory_1_memory-0-0x00000000]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590000000000000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3E90DCEC8F877A6B. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-5-0xA0000005
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.2.41:11111:149226787 sys.svc.stream.agent/10.244.2.41:11111@149226787+Memory_3_memory-5-0xA0000005]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb4887559050000a005000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3E90DCEC8F877A68. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-3-0x60000003
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.9.67:11111:149185698 sys.svc.stream.agent/10.244.9.67:11111@149185698+Memory_3_memory-3-0x60000003]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590300006003000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #6E969F91DB24842B. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-2-0x40000002
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.9.67:11111:149185698 sys.svc.stream.agent/10.244.9.67:11111@149185698+Memory_7_memory-2-0x40000002]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590200004002000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #6E969F91DB24842A. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-4-0x80000004
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.9.67:11111:149185698 sys.svc.stream.agent/10.244.9.67:11111@149185698+Memory_4_memory-4-0x80000004]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590400008004000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #6E969F91DB248437. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241

Re-addressed by the current identity function: this incident inherited 3a8dfa6d2e5c4d2d (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Opened automatically from Admin/_LogIncident/2130b40397e707aa. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 56050d06f832b3ce: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
5h 10m
PR unite (30g)
980

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.