Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Grain-directory lookup routed to departed silo S10.244.3.115 fails message-hub placement (SiloUnavailableException)

Aperta
  • Qualcosa non va?
#7,002 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp, kubernetes
Ambito
backend, distributed-systems

Direzione di ricerca

Start by checking Kubernetes events and the pod lifecycle for the silo at 10.244.3.115 around 2026-10-20 14:15 UTC. Then read the surviving silos’ MembershipTableManager and MembershipAgent logs for the same window, alongside the linked #5336 incident and the listed Orleans incidents. Determine whether membership stayed stale after the silo departed or a graceful drain was in progress; the cause and whether the messages were delivered remain unestablished.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:L

Observed

Two MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage requests from portal pod memex-portal-deployment-85f94799c4-lw5sv (namespace memex-cloud, client silo S10.244.5.170:11111:151508593) — targets messagehub/FractalStars/_Entitlements/sebastien.duble and messagehub/Reinsurance/ModelledReinsuranceSection — could not be placed. The failing call was Orleans.GrainDirectory.IDhtGrainDirectory.LookupAsync (via DhtGrainLocator.Lookup → PlacementService.PlacementWorker.ExecutePlacementAsync), addressed to silo S10.244.3.115:11111:151508771 (sys.svc.dir.mem, i.e. the silo owning that grain-directory partition), which raised Orleans.Runtime.SiloUnavailableException ("The target silo became unavailable"). The Orleans placement retry pipeline (Orleans.Placement/(null)/Retry) had already exhausted attempt 3 (≈16 ms per attempt) before the failure surfaced — captured at the Polly log site in sibling incident 7a44136a5a9958e3 (same pod, same instant, same target silo).

Where and when

Two occurrences within 21 ms, one pod, one burst — not a steady rate. This burst is the tail of a single event: one second earlier (14:15:32.53–56) the same pod logged OrleansMessageRejectionException / ConnectionRefused while connecting to the same silo 10.244.3.115:11111 for message-hub deliveries — the live edge of incident f13960a22390eb3a, filed as Systemorph/MeshWeaver#5336, whose lastSeen is 14:15:32.56, immediately before this burst. So within one window: direct connections to 10.244.3.115 were refused (nothing listening), and directory lookups routed to it still failed after retries.

Impact

Minimal as logged: two message-hub deliveries, one pod, sub-second window. The wider concern is the shape, not the count — while the directory partition owner sys.svc.dir.mem/10.244.3.115 is unreachable, every grain whose directory entry hashes to that partition cannot be placed by any client, not only these two message-hub deliveries. Whether that window lasted one second (as logged here) or longer is not established.

Not established

  • Whether silo 10.244.3.115 crashed, was OOM-killed, or rolled normally.
  • Whether membership/directory eventually converged on its own or the condition persisted beyond this burst (the ConnectionRefused parent incident #5336 has recurred since 2026-10-01).
  • Whether the two undelivered messages reached their message-hub grains afterwards.

Likely duplicate / related

This is very probably the same event as Systemorph/MeshWeaver#5336 (stale membership serving a departed silo 10.244.3.115), observed at a different failure surface: #5336 captures the direct send to the departed silo being refused (ConnectionRefused); this incident captures the grain-directory lookup routed to that silo failing inside placement (SiloUnavailableException after retry exhaustion). One ticket may well cover both; opening separately so the directory-lookup surface is visible on the record.

Candidate causes and the reading that separates them

  1. Stale membership / directory entry after the silo departed (the #5336 family): the process at 10.244.3.115:11111 was gone (ConnectionRefused says nothing is listening) but the cluster still routed directory lookups to it. Predicts rejection bursts coincide with the pod's death, and stop once membership converges. Reading that confirms: Kubernetes events for the pod at 10.244.3.115 around 14:15:30 UTC on 2026-10-20 (restart/roll/eviction), and whether MembershipTableManager/MembershipAgent logs on the surviving silos show a late eviction or clustering-table trouble at that time (the root cause recorded in #2307's episode was the clustering table being unreachable).
  2. Expected transient during silo shutdown/drain — the same fingerprint family as suppressed incident ca34cb341a76fbb8, where in-flight requests to a draining silo fail and the runtime re-places them. This shape differs from that one (there the message said "Silo … is shutting down" and placement re-tried; here the endpoint refused connections and the Polly retry pipeline exhausted all 3 attempts before surfacing), which argues against — but does not exclude — ordinary drain. Reading that separates: shutdown/drain logs from the process at 10.244.3.115 in the seconds before 14:15:33; a graceful drain predicts "Silo is shutting down" messages and successful re-placement, their absence predicts cause 1.

Where to look

  • Kubernetes events and pod lifecycle for the pod holding 10.244.3.115 (replicaset generation active at 2026-10-20 14:15 UTC).
  • Orleans membership logs (MembershipTableManager, MembershipAgent) on surviving memex-cloud silos around 14:15:30–14:16 UTC.
  • Sibling incidents from the same window: f13960a22390eb3a (#5336, ConnectionRefused to the same silo), 7a44136a5a9958e3 (Polly retry exhaustion), and the suppressed drain fingerprint ca34cb341a76fbb8 for the contrast shape.
Evidence
Fingerprint 31612dc1df151a5b
Category Orleans.Messaging
Severity Error
Exception Orleans.Runtime.SiloUnavailableException
Namespace memex-cloud
Pods memex-portal-deployment-85f94799c4-lw5sv
Occurrences 2
First seen 2026-10-20 14:15:33Z
Last seen 2026-10-20 14:15:33Z
Routing not determined — no configured route matches the category Orleans.Messaging. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-20 14:15:33Z memex-portal-deployment-85f94799c4-lw5sv fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.170:11111:151508593 sys.client/hosted-10.244.5.170:11111@151508593]->[ messagehub/FractalStars/_Entitlements/sebastien.duble] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #18AE7F269116D234
      Orleans.Runtime.SiloUnavailableException: The target silo became unavailable for message: Request [S10.244.5.170:11111:151508593 sys.client/hosted-10.244.5.170:11111@151508593]->[S10.244.3.115:11111:151508771 sys.svc.dir.mem/10.244.3.115:11111@151508771] Orleans.GrainDirectory.IDhtGrainDirectory.LookupAsync(Orleans.Runtime.GrainId, System.Int32) #18AE7F269116D949. See https://aka.ms/orleans-troubleshooting for troubleshooting help.
         at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 239
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans…[truncated]
2026-10-20 14:15:33Z memex-portal-deployment-85f94799c4-lw5sv fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.170:11111:151508593 sys.client/hosted-10.244.5.170:11111@151508593]->[ messagehub/Reinsurance/ModelledReinsuranceSection] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #18AE7F269116D5FA
      Orleans.Runtime.SiloUnavailableException: The target silo became unavailable for message: Request [S10.244.5.170:11111:151508593 sys.client/hosted-10.244.5.170:11111@151508593]->[S10.244.3.115:11111:151508771 sys.svc.dir.mem/10.244.3.115:11111@151508771] Orleans.GrainDirectory.IDhtGrainDirectory.LookupAsync(Orleans.Runtime.GrainId, System.Int32) #18AE7F269116D94D. See https://aka.ms/orleans-troubleshooting for troubleshooting help.
         at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 239
         at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
         at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Run…[truncated]

Opened automatically from Admin/_LogIncident/31612dc1df151a5b. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 8e3325d86cf4e6a7: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
6h
PR unite (30g)
967

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.