Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Memory stream queue dequeues time out when target silo is unresponsive; pulling agent retries without recovering

Đã đóng
  • Có gì không ổn?
#5,704 30 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
35/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
csharp
Lĩnh vực
backend

Hướng nghiên cứu

Read PersistentStreamPullingAgent.ReadFromQueue and MemoryAdapterReceiver.GetQueueMessagesAsync in src/Orleans.Streaming/, then investigate the health of silo 10.244.3.152 during the reported timeout window and cross-reference the sibling incidents listed. The issue reports a transient, self-recovered cluster-wide symptom and does not establish a code change; done would require identifying a root cause and an appropriate fix rather than merely raising the timeout.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug sev:M

What is failing
The Orleans persistent-stream pulling agents on the portal pods cannot read from any of their memory stream queues: every IMemoryStreamQueueGrain.Dequeue request sent to the queue grains on the target silo 10.244.3.152:11111 exceeds the 30-second default response timeout. The retry wrapper (PersistentStreamPullingAgent.ReadFromQueue via AsyncExecutorWithRetries) logs and retries, but the requests keep timing out for the whole window, so stream delivery through all eight memory queues stalls.

Probable cause
Not a defect in the memory stream adapter itself — confidence high that this is downstream of an unresponsive or overloaded silo. Every sample targets the same silo address and the same memorystreamqueue grain placement, while two different pods (and all queue ids 0–7) fail identically and simultaneously; that is the signature of the target silo's grain scheduler being stalled or the process saturated, not of per-queue corruption. This incident is one of a cluster-wide family of System.TimeoutException incidents in the same namespace over 2026-09-22 → 09-24 (Messaging "Failed to address message", RoutingGrain directed-delivery failures, ClientDirectory routing-table publish, ActivityTracking, DataContext hub initialization timeouts — e.g. Admin/_LogIncident/f7d8f9982b20cce5, 254804a0aad8b746, af71d14af5a4d50c), which strengthens the read that one silo or the cluster as a whole was degraded. Root cause of that degradation is not determinable from this evidence alone.

Impact
21 occurrences across both portal pods inside a single ~80-second window, then it stops. Stream delivery to subscribers on those queues was stalled for that window; self-recovered. Given the sibling incidents, treat this as one symptom of the larger timeout storm rather than an isolated defect.

Where to look
Start from Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue and Orleans.Providers.MemoryAdapterReceiver.GetQueueMessagesAsync (Orleans framework source, src/Orleans.Streaming/), then investigate the health of the silo at 10.244.3.152 during the window — grain-scheduler queue depth, GC pauses, CPU/memory — and cross-reference with the sibling timeout incidents above rather than fixing them one by one. If the 30s dequeue timeout proves too tight under load, raising the stream-pulling agent's execution timeout is a workaround, not the fix.

Evidence
Fingerprint 2130b40397e707aa
Category Orleans.Streams.Memory
Severity Error
Exception System.TimeoutException
Namespace memex
Pods memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-56fbdcd48f-sz5st
Occurrences 21
First seen 2026-09-24 06:26:15Z
Last seen 2026-09-24 06:27:38Z
Routing not determined — no configured route matches the category Orleans.Streams.Memory. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-09-24 06:27:38Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-0-0x00000000
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.2.41:11111:149226787 sys.svc.stream.agent/10.244.2.41:11111@149226787+Memory_1_memory-0-0x00000000]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590000000000000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3E90DCEC8F877A6B. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-5-0xA0000005
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.2.41:11111:149226787 sys.svc.stream.agent/10.244.2.41:11111@149226787+Memory_3_memory-5-0xA0000005]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb4887559050000a005000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #3E90DCEC8F877A68. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-3-0x60000003
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.9.67:11111:149185698 sys.svc.stream.agent/10.244.9.67:11111@149185698+Memory_3_memory-3-0x60000003]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590300006003000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #6E969F91DB24842B. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-2-0x40000002
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.9.67:11111:149185698 sys.svc.stream.agent/10.244.9.67:11111@149185698+Memory_7_memory-2-0x40000002]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590200004002000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #6E969F91DB24842A. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241
2026-09-24 06:27:38Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Streams.Memory[103312]
      Exception while retrying the 2th time reading from queue memory-4-0x80000004
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.9.67:11111:149185698 sys.svc.stream.agent/10.244.9.67:11111@149185698+Memory_4_memory-4-0x80000004]->[S10.244.3.152:11111:149145938 memorystreamqueue/be8eb06fb48875590400008004000000] Orleans.Providers.IMemoryStreamQueueGrain.Dequeue(System.Int32) #6E969F91DB248437. 
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Providers.MemoryAdapterReceiver`1.GetQueueMessagesAsync(Int32 maxCount) in /_/src/Orleans.Streaming/MemoryStreams/MemoryAdapterReceiver.cs:line 45
         at Orleans.Streams.PersistentStreamPullingAgent.ReadFromQueue(QueueId myQueueId, IQueueAdapterReceiver rcvr, Int32 maxCacheAddCount, CancellationToken cancellationToken) in /_/src/Orleans.Streaming/PersistentStreams/PersistentStreamPullingAgent.cs:line 572
         at Orleans.Internal.AsyncExecutorWithRetries.ExecuteWithRetriesHelper[T](Func`2 function, Int32 maxNumSuccessTries, Int32 maxNumErrorTries, TimeSpan maxExecutionTime, DateTime startExecutionTime, Func`3 retryValueFilter, Func`3 retryExceptionFilter, IBackoffProvider onSuccessBackOff, IBackoffProvider onErrorBackOff, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/AsyncExecutorWithRetries.cs:line 241

Re-addressed by the current identity function: this incident inherited 3a8dfa6d2e5c4d2d (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Opened automatically from Admin/_LogIncident/2130b40397e707aa. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 56050d06f832b3ce: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Ngôn ngữ chính
C#
Star
12
Fork
5
Merge trung bình
5 giờ 51 phút
Pull request đã merge (30 ngày)
970

Chuẩn bị môi trường

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của Systemorph/MeshWeaver

Tất cả issue của Systemorph/MeshWeaver

Issue tương tự

Thêm issue về C#

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.