ClientDirectory client-routing-table publish to silo 10.244.6.246 timed out during a ~70 s window in which that silo answered no inter-silo traffic

Aperta
  • Qualcosa non va?
#7,815 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
15/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
csharp, kubernetes
Ambito
distributed-systems

Direzione di ricerca

The failing call is PublishUpdates() in Orleans.Runtime/GrainDirectory/ClientDirectory.cs (line 592, upstream), but the code there is not the cause. Start with the pod holding 10.244.6.246 in memex-cloud and the membership table history for epoch 152555378 around 21:32-21:34Z. Done means deciding whether the silo restarted (cause 1) or stalled while running (cause 2), and attaching that finding to the sibling incidents.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:L

Observed

Orleans' ClientDirectory on hosted client 10.244.22.30:11111:152555210 (pod memex-portal-deployment-c94c4ff44-x9ws2) failed to publish the client routing table to silo S10.244.6.246:11111:152555378: the IRemoteClientDirectory.OnUpdateClientRoutes call to that silo's sys.svc.dir.client system target did not return within the 30 s response timeout and threw System.TimeoutException, caught in ClientDirectory.PublishUpdates(). The routing update therefore did not reach that silo in this publish cycle.

Where and when

One occurrence, at 2026-11-01T21:33:24Z, on one pod. It sits inside a ~70-second window in which every observed inter-silo path to S10.244.6.246:11111:152555378 failed, from multiple independent senders:

  • 21:32:49–21:33:55 — RoutingGrain directed deliveries to IPodHubGrain grains on that silo timed out from two different pods (incident {path}, filed as #5337)
  • 21:32:56–21:33:34 — GrainCallCancellationManager calls to that silo's sys.svc.canceler timed out from two different silos (incident {path}, filed as #6392)
  • 21:32:58–21:33:56 — a newly joining silo (pod memex-portal-deployment-c94c4ff44-9srqr) failed its membership connectivity validation ping to that silo, attempts #1–#4, while the other two active silos answered (incident {path})

This is a cluster-visible event on one silo, not an isolated defect in the ClientDirectory path: the publish failed for the same reason the pings, canceler calls and deliveries did — the target silo was not answering.

Impact

Judged from the numbers: one occurrence of this log site, one pod, no recurrence before or since on this fingerprint. The immediate consequence is one client-routing-table update not reaching one silo. No user-visible failure is evidenced on this incident; the correlated delivery failures on sibling sites (#5337) show messages to pods on that silo did fail in the same window, but those are tracked on their own tickets.

Not established

  • The cause. Nothing on this incident says why S10.244.6.246:11111:152555378 stopped answering.
  • Whether that silo was shutting down, had died, or was alive but stalled.
  • Whether the membership table eventually declared it dead, and whether the joining silo completed its join.
  • Whether the client routing table was successfully re-published to that silo (or its replacement) after the window.
  • Whether this ClientDirectory log site fires outside this window; only this one occurrence is on record.

Candidate causes and the reading that separates them

  1. The silo departed or was terminated (deploy, scale-down, pod kill) while still listed Active in the membership table. Predicts: the Kubernetes pod holding IP 10.244.6.246 shows {path} around 21:32–21:34Z, and the membership table shows a status transition for S10.244.6.246:11111:152555378 shortly after. This would also align with #5300's theme of stale Active entries after non-graceful silo termination.
  2. The silo was alive but stalled (blocked runtime, GC pause, overload) for the duration of the window. Predicts: the pod at 10.244.6.246 has no restart in that window but its own logs show a {path} or overload signatures at 21:32–21:34Z, and it resumed answering afterwards.

The reading that separates them: the pod lifecycle and logs of the pod holding IP 10.244.6.246 around 21:32–21:34Z, plus the membership-table status history for silo epoch 152555378. A restart confirms cause 1; continuous logs with a stall confirm cause 2. Whichever it is, the same reading explains the four sibling incidents in this window and should be attached to all of them.

Where to look

  • Orleans.Runtime.GrainDirectory.ClientDirectory.PublishUpdates() — the log site (upstream path {path}:592; the Orleans framework source is not present in this mesh).
  • The pod behind IP 10.244.6.246 in the memex-cloud deployment at 2026-11-01T21:32–21:34Z.
  • Sibling incidents of the same window, for correlation only: {path} (membership ping failures, suppressed as covered by #5300), 187c938979420f7d (#6392), c8a606769e85680c (#5337). Not a duplicate of any of these: distinct log site, distinct subsystem — but almost certainly one shared underlying event.
Evidence
Fingerprint 6b44a7a33abb8555
Category Orleans.Runtime.GrainDirectory.ClientDirectory
Severity Error
Exception System.TimeoutException
Namespace memex-cloud 1 occurrence(s), 2026-11-01 21:33:24Z → 2026-11-01 21:33:24Z; pods memex-portal-deployment-c94c4ff44-x9ws2
Occurrences 1
First seen 2026-11-01 21:33:24Z
Last seen 2026-11-01 21:33:24Z
Routing not determined — no configured route matches the category Orleans.Runtime.GrainDirectory.ClientDirectory. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-11-01 21:33:24Z memex-portal-deployment-c94c4ff44-x9ws2 fail: Orleans.Runtime.GrainDirectory.ClientDirectory[0]
      Exception publishing client routing table to silo S10.244.6.246:11111:152555378
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.22.30:11111:152555210 {path}:11111@152555210]->[S10.244.6.246:11111:152555378 {path}:11111@152555378] Orleans.Runtime.GrainDirectory.IRemoteClientDirectory.OnUpdateClientRoutes(System.Collections.Immutable.ImmutableDictionary`2[Orleans.Runtime.SiloAddress,System.ValueTuple`2[System.Collections.Immutable.ImmutableHashSet`1[Orleans.Runtime.GrainId],System.Int64]], System.Threading.CancellationToken) #13BC8675FB120244. 
         at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 100
         at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.ClientDirectory.PublishUpdates() in /_/src/Orleans.Runtime/GrainDirectory/ClientDirectory.cs:line 592

This log site's previous ticket, https://github.com/Systemorph/MeshWeaver/issues/2906, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

Opened automatically from Admin/_LogIncident/6b44a7a33abb8555. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site fcd58a9e3a377731: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
5h 11m
PR unite (30g)
978

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.