ClientDirectory client-routing-table publish to silo 10.244.6.246 timed out during a ~70 s window in which that silo answered no inter-silo traffic

Open
  • Is something wrong?
#7,815 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
15/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
csharp, kubernetes
Domain
distributed-systems

Research direction

The failing call is PublishUpdates() in Orleans.Runtime/GrainDirectory/ClientDirectory.cs (line 592, upstream), but the code there is not the cause. Start with the pod holding 10.244.6.246 in memex-cloud and the membership table history for epoch 152555378 around 21:32-21:34Z. Done means deciding whether the silo restarted (cause 1) or stalled while running (cause 2), and attaching that finding to the sibling incidents.

Written by the indexing model from the issue text.

Description

bug sev:L

Observed

Orleans' ClientDirectory on hosted client 10.244.22.30:11111:152555210 (pod memex-portal-deployment-c94c4ff44-x9ws2) failed to publish the client routing table to silo S10.244.6.246:11111:152555378: the IRemoteClientDirectory.OnUpdateClientRoutes call to that silo's sys.svc.dir.client system target did not return within the 30 s response timeout and threw System.TimeoutException, caught in ClientDirectory.PublishUpdates(). The routing update therefore did not reach that silo in this publish cycle.

Where and when

One occurrence, at 2026-11-01T21:33:24Z, on one pod. It sits inside a ~70-second window in which every observed inter-silo path to S10.244.6.246:11111:152555378 failed, from multiple independent senders:

  • 21:32:49–21:33:55 — RoutingGrain directed deliveries to IPodHubGrain grains on that silo timed out from two different pods (incident {path}, filed as #5337)
  • 21:32:56–21:33:34 — GrainCallCancellationManager calls to that silo's sys.svc.canceler timed out from two different silos (incident {path}, filed as #6392)
  • 21:32:58–21:33:56 — a newly joining silo (pod memex-portal-deployment-c94c4ff44-9srqr) failed its membership connectivity validation ping to that silo, attempts #1–#4, while the other two active silos answered (incident {path})

This is a cluster-visible event on one silo, not an isolated defect in the ClientDirectory path: the publish failed for the same reason the pings, canceler calls and deliveries did — the target silo was not answering.

Impact

Judged from the numbers: one occurrence of this log site, one pod, no recurrence before or since on this fingerprint. The immediate consequence is one client-routing-table update not reaching one silo. No user-visible failure is evidenced on this incident; the correlated delivery failures on sibling sites (#5337) show messages to pods on that silo did fail in the same window, but those are tracked on their own tickets.

Not established

  • The cause. Nothing on this incident says why S10.244.6.246:11111:152555378 stopped answering.
  • Whether that silo was shutting down, had died, or was alive but stalled.
  • Whether the membership table eventually declared it dead, and whether the joining silo completed its join.
  • Whether the client routing table was successfully re-published to that silo (or its replacement) after the window.
  • Whether this ClientDirectory log site fires outside this window; only this one occurrence is on record.

Candidate causes and the reading that separates them

  1. The silo departed or was terminated (deploy, scale-down, pod kill) while still listed Active in the membership table. Predicts: the Kubernetes pod holding IP 10.244.6.246 shows {path} around 21:32–21:34Z, and the membership table shows a status transition for S10.244.6.246:11111:152555378 shortly after. This would also align with #5300's theme of stale Active entries after non-graceful silo termination.
  2. The silo was alive but stalled (blocked runtime, GC pause, overload) for the duration of the window. Predicts: the pod at 10.244.6.246 has no restart in that window but its own logs show a {path} or overload signatures at 21:32–21:34Z, and it resumed answering afterwards.

The reading that separates them: the pod lifecycle and logs of the pod holding IP 10.244.6.246 around 21:32–21:34Z, plus the membership-table status history for silo epoch 152555378. A restart confirms cause 1; continuous logs with a stall confirm cause 2. Whichever it is, the same reading explains the four sibling incidents in this window and should be attached to all of them.

Where to look

  • Orleans.Runtime.GrainDirectory.ClientDirectory.PublishUpdates() — the log site (upstream path {path}:592; the Orleans framework source is not present in this mesh).
  • The pod behind IP 10.244.6.246 in the memex-cloud deployment at 2026-11-01T21:32–21:34Z.
  • Sibling incidents of the same window, for correlation only: {path} (membership ping failures, suppressed as covered by #5300), 187c938979420f7d (#6392), c8a606769e85680c (#5337). Not a duplicate of any of these: distinct log site, distinct subsystem — but almost certainly one shared underlying event.
Evidence
Fingerprint 6b44a7a33abb8555
Category Orleans.Runtime.GrainDirectory.ClientDirectory
Severity Error
Exception System.TimeoutException
Namespace memex-cloud 1 occurrence(s), 2026-11-01 21:33:24Z → 2026-11-01 21:33:24Z; pods memex-portal-deployment-c94c4ff44-x9ws2
Occurrences 1
First seen 2026-11-01 21:33:24Z
Last seen 2026-11-01 21:33:24Z
Routing not determined — no configured route matches the category Orleans.Runtime.GrainDirectory.ClientDirectory. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-11-01 21:33:24Z memex-portal-deployment-c94c4ff44-x9ws2 fail: Orleans.Runtime.GrainDirectory.ClientDirectory[0]
      Exception publishing client routing table to silo S10.244.6.246:11111:152555378
      System.TimeoutException: Response did not arrive on time in 00:00:30 for message: Request [S10.244.22.30:11111:152555210 {path}:11111@152555210]->[S10.244.6.246:11111:152555378 {path}:11111@152555378] Orleans.Runtime.GrainDirectory.IRemoteClientDirectory.OnUpdateClientRoutes(System.Collections.Immutable.ImmutableDictionary`2[Orleans.Runtime.SiloAddress,System.ValueTuple`2[System.Collections.Immutable.ImmutableHashSet`1[Orleans.Runtime.GrainId],System.Int64]], System.Threading.CancellationToken) #13BC8675FB120244. 
         at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 100
         at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainDirectory.ClientDirectory.PublishUpdates() in /_/src/Orleans.Runtime/GrainDirectory/ClientDirectory.cs:line 592

This log site's previous ticket, https://github.com/Systemorph/MeshWeaver/issues/2906, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

Opened automatically from Admin/_LogIncident/6b44a7a33abb8555. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site fcd58a9e3a377731: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
5h 10m
Merged PRs (30d)
980

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.