Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Grain-directory lookup routed to departed silo S10.244.3.115 fails message-hub placement (SiloUnavailableException)

Open
  • Is something wrong?
#7,003 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp, kubernetes
Domain
backend, infrastructure

Research direction

Start by correlating Kubernetes events and pod lifecycle for the silo at 10.244.3.115 with Orleans MembershipTableManager and MembershipAgent logs on surviving silos around 14:15:30–14:16 UTC on 2026-10-20. Compare those logs with incident #5336 and the listed drain fingerprint to distinguish stale membership from shutdown. Done means the cause and whether membership converged are established; the payload does not identify a code change.

Written by the indexing model from the issue text.

Description

bug sev:L

Observed

Two MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage requests from portal pod memex-portal-deployment-85f94799c4-lw5sv (namespace memex-cloud, client silo S10.244.5.170:11111:151508593) — targets messagehub/FractalStars/_Entitlements/sebastien.duble and messagehub/Reinsurance/ModelledReinsuranceSection — could not be placed. The failing call was Orleans.GrainDirectory.IDhtGrainDirectory.LookupAsync (via DhtGrainLocator.Lookup → PlacementService.PlacementWorker.ExecutePlacementAsync), addressed to silo S10.244.3.115:11111:151508771 (sys.svc.dir.mem, i.e. the silo owning that grain-directory partition), which raised Orleans.Runtime.SiloUnavailableException ("The target silo became unavailable"). The Orleans placement retry pipeline (Orleans.Placement/(null)/Retry) had already exhausted attempt 3 (≈16 ms per attempt) before the failure surfaced — captured at the Polly log site in sibling incident 7a44136a5a9958e3 (same pod, same instant, same target silo).

Where and when

Two occurrences within 21 ms, one pod, one burst — not a steady rate. This burst is the tail of a single event: one second earlier (14:15:32.53–56) the same pod logged OrleansMessageRejectionException / ConnectionRefused while connecting to the same silo 10.244.3.115:11111 for message-hub deliveries — the live edge of incident f13960a22390eb3a, filed as Systemorph/MeshWeaver#5336, whose lastSeen is 14:15:32.56, immediately before this burst. So within one window: direct connections to 10.244.3.115 were refused (nothing listening), and directory lookups routed to it still failed after retries.

Impact

Minimal as logged: two message-hub deliveries, one pod, sub-second window. The wider concern is the shape, not the count — while the directory partition owner sys.svc.dir.mem/10.244.3.115 is unreachable, every grain whose directory entry hashes to that partition cannot be placed by any client, not only these two message-hub deliveries. Whether that window lasted one second (as logged here) or longer is not established.

Not established

  • Whether silo 10.244.3.115 crashed, was OOM-killed, or rolled normally.
  • Whether membership/directory eventually converged on its own or the condition persisted beyond this burst (the ConnectionRefused parent incident #5336 has recurred since 2026-10-01).
  • Whether the two undelivered messages reached their message-hub grains afterwards.

Likely duplicate / related

This is very probably the same event as Systemorph/MeshWeaver#5336 (stale membership serving a departed silo 10.244.3.115), observed at a different failure surface: #5336 captures the direct send to the departed silo being refused (ConnectionRefused); this incident captures the grain-directory lookup routed to that silo failing inside placement (SiloUnavailableException after retry exhaustion). One ticket may well cover both; opening separately so the directory-lookup surface is visible on the record.

Candidate causes and the reading that separates them

  1. Stale membership / directory entry after the silo departed (the #5336 family): the process at 10.244.3.115:11111 was gone (ConnectionRefused says nothing is listening) but the cluster still routed directory lookups to it. Predicts rejection bursts coincide with the pod's death, and stop once membership converges. Reading that confirms: Kubernetes events for the pod at 10.244.3.115 around 14:15:30 UTC on 2026-10-20 (restart/roll/eviction), and whether MembershipTableManager/MembershipAgent logs on the surviving silos show a late eviction or clustering-table trouble at that time (the root cause recorded in #2307's episode was the clustering table being unreachable).
  2. Expected transient during silo shutdown/drain — the same fingerprint family as suppressed incident ca34cb341a76fbb8, where in-flight requests to a draining silo fail and the runtime re-places them. This shape differs from that one (there the message said "Silo … is shutting down" and placement re-tried; here the endpoint refused connections and the Polly retry pipeline exhausted all 3 attempts before surfacing), which argues against — but does not exclude — ordinary drain. Reading that separates: shutdown/drain logs from the process at 10.244.3.115 in the seconds before 14:15:33; a graceful drain predicts "Silo is shutting down" messages and successful re-placement, their absence predicts cause 1.

Where to look

  • Kubernetes events and pod lifecycle for the pod holding 10.244.3.115 (replicaset generation active at 2026-10-20 14:15 UTC).
  • Orleans membership logs (MembershipTableManager, MembershipAgent) on surviving memex-cloud silos around 14:15:30–14:16 UTC.
  • Sibling incidents from the same window: f13960a22390eb3a (#5336, ConnectionRefused to the same silo), 7a44136a5a9958e3 (Polly retry exhaustion), and the suppressed drain fingerprint ca34cb341a76fbb8 for the contrast shape.
Evidence
Fingerprint 31612dc1df151a5b
Category Orleans.Messaging
Severity Error
Exception Orleans.Runtime.SiloUnavailableException
Namespace memex-cloud
Pods memex-portal-deployment-85f94799c4-lw5sv
Occurrences 2
First seen 2026-10-20 14:15:33Z
Last seen 2026-10-20 14:15:33Z
Routing not determined — no configured route matches the category Orleans.Messaging. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-20 14:15:33Z memex-portal-deployment-85f94799c4-lw5sv fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.170:11111:151508593 sys.client/hosted-10.244.5.170:11111@151508593]->[ messagehub/FractalStars/_Entitlements/sebastien.duble] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #18AE7F269116D234
      Orleans.Runtime.SiloUnavailableException: The target silo became unavailable for message: Request [S10.244.5.170:11111:151508593 sys.client/hosted-10.244.5.170:11111@151508593]->[S10.244.3.115:11111:151508771 sys.svc.dir.mem/10.244.3.115:11111@151508771] Orleans.GrainDirectory.IDhtGrainDirectory.LookupAsync(Orleans.Runtime.GrainId, System.Int32) #18AE7F269116D949. See https://aka.ms/orleans-troubleshooting for troubleshooting help.
         at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 239
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans…[truncated]
2026-10-20 14:15:33Z memex-portal-deployment-85f94799c4-lw5sv fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.170:11111:151508593 sys.client/hosted-10.244.5.170:11111@151508593]->[ messagehub/Reinsurance/ModelledReinsuranceSection] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #18AE7F269116D5FA
      Orleans.Runtime.SiloUnavailableException: The target silo became unavailable for message: Request [S10.244.5.170:11111:151508593 sys.client/hosted-10.244.5.170:11111@151508593]->[S10.244.3.115:11111:151508771 sys.svc.dir.mem/10.244.3.115:11111@151508771] Orleans.GrainDirectory.IDhtGrainDirectory.LookupAsync(Orleans.Runtime.GrainId, System.Int32) #18AE7F269116D94D. See https://aka.ms/orleans-troubleshooting for troubleshooting help.
         at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 239
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Run…[truncated]

Opened automatically from Admin/_LogIncident/31612dc1df151a5b. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 8e3325d86cf4e6a7: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
6h 29m
Merged PRs (30d)
914

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.