Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Memory stream queue dequeue times out (30s) against a churn-affected remote silo

Chiusa
  • Qualcosa non va?
#6,431 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
18/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
distributed-systems, stream-processing

Direzione di ricerca

The failing call is IMemoryStreamQueueGrain.Dequeue made from GetQueueMessagesAsync in src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs (line 54 in the logs), but that file lives in the Orleans framework, not this repository, and the issue itself attributes the timeouts to cluster churn rather than a code defect. Start by reading the receiver's poll loop and deciding whether the timeout belongs at the call site or in the silo-churn handling. Done would be a maintainer decision on whether to shorten the timeout or fix drain behaviour, backed by the linked churn incidents.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:L

What is failing

On the memex-cloud deployment, the Orleans memory-2 memory-stream adapter's receive loop (MemoryAdapterReceiver.GetQueueMessagesAsync) occasionally fails its IMemoryStreamQueueGrain.Dequeue call: the response from the memorystreamqueue grain hosted on a different silo does not arrive within the 30-second Orleans response timeout.

Probable cause

Most likely transient cluster churn/degradation, not a code defect in the dequeue path itself. Evidence:

  • The dequeue target is a remote silo (agent sys.svc.stream.agent calling a memorystreamqueue grain on another node), so the call is sensitive to that silo's health.
  • The two failing pods belong to two different deployment generations (-748f7f4577 and -85587f8869), consistent with a rollout in progress when the samples were taken.
  • The same log site (MemoryAdapterReceiver on memory-2) has an entire family of sibling incidents around pod rolls since mid-September — SiloUnavailableException (e.g. Admin/_LogIncident/roll-churn-5239e1d924425fbc, Admin/_LogIncident/1d487661c5db40b7), OrleansMessageRejectionException, and OperationCanceledException — sharing this incident's site fold. That points to the target silo being wedged or shutting down mid-roll, so the dequeue call burns its full timeout instead of failing fast.

Confidence: moderate-high on the mechanism (slow/absent response from a churn-affected silo), lower on whether the timeout itself is worth fixing at the call site. If this churn-timeout family keeps growing, the underlying issue may be that the deployment rolls do not drain memory-stream agents cleanly.

Impact

Low. Three occurrences over three days across two pods; the receiver retries the dequeue on its next poll cycle, so the effect is a delayed poll rather than lost messages. No evidence of user-visible streaming breakage tied to these specific timeouts.

Where to look

  • Orleans framework source: src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs, GetQueueMessagesAsync (the failing call is logged there; samples point at line 54 in the Orleans 10 build).
  • The grain interface: Orleans.Providers.IMemoryStreamQueueGrain.Dequeue — consider whether its timeout should be shorter than the default 30s response timeout for an empty-queue poll.
  • Our streaming architecture context: Workspace References & Streaming.
  • The churn-timeout family on this log site for correlated investigation.
Evidence
Fingerprint ef4a6f54e5296080
Category Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-2`
Severity Error
Exception System.TimeoutException
Namespace memex-cloud
Pods memex-portal-deployment-748f7f4577-pjrjp, memex-portal-deployment-85587f8869-5j9gx
Occurrences 3
First seen 2026-10-07 09:53:56Z
Last seen 2026-10-10 08:25:48Z
Routing not determined — no configured route matches the category Orleans.Providers.MemoryAdapterReceiver1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-2`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-07 09:53:56Z memex-portal-deployment-748f7f4577-pjrjp fail: Orleans.Providers.MemoryAdapterReceiver`1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-2[200402]
      Exception thrown in MemoryAdapterFactory.GetQueueMessagesAsync.
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.215:11111:150368445 sys.svc.stream.agent/10.244.5.215:11111@150368445+Memory_2_memory-2-0x40000002]->[S10.244.2.27:11111:150368365 memorystreamqueue/be8eb06fb48875590200004002000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32, System.Threading.CancellationToken) #0738373D83D38639. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 54
2026-10-07 09:54:37Z memex-portal-deployment-748f7f4577-pjrjp fail: Orleans.Providers.MemoryAdapterReceiver`1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-2[200402]
      Exception thrown in MemoryAdapterFactory.GetQueueMessagesAsync.
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.5.215:11111:150368445 sys.svc.stream.agent/10.244.5.215:11111@150368445+Memory_2_memory-2-0x40000002]->[S10.244.2.27:11111:150368365 memorystreamqueue/be8eb06fb48875590200004002000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32, System.Threading.CancellationToken) #0738373D83D3199A. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 54
2026-10-10 08:25:48Z memex-portal-deployment-85587f8869-5j9gx fail: Orleans.Providers.MemoryAdapterReceiver`1[[Orleans.Providers.DefaultMemoryMessageBodySerializer, Orleans.Streaming, Version=10.0.0.0, Culture=neutral, PublicKeyToken=null]].Memory.memory-2[200402]
      Exception thrown in MemoryAdapterFactory.GetQueueMessagesAsync.
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.6.128:11111:150624604 sys.svc.stream.agent/10.244.6.128:11111@150624604+Memory_2_memory-2-0x40000002]->[S10.244.5.66:11111:150623330 memorystreamqueue/be8eb06fb48875590200004002000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32, System.Threading.CancellationToken) #0E50769F7196FCD3. 
         at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 236
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 54

Opened automatically from Admin/_LogIncident/ef4a6f54e5296080. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 2e42ae84e51bb7ce: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
6h 37m
PR unite (30g)
913

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.