Orleans ActivationMigrationManager times out migrating single grain activations to two target silos on memex-cloud

Aperta
  • Qualcosa non va?
#7,861 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
distributed-systems

Direzione di ricerca

Start with Orleans.Runtime.ActivationMigrationManager.PumpMigrationQueue at ActivationMigrationManager.cs:734, then read its informational log lines just before the failures. Compare the reporting pod's migration activity with the target silos' LocalSiloHealthMonitor and membership logs, and read incidents #5334, #5388, and #7002 for the same minute. Done means identifying a supported cause or documenting what evidence is still missing; this issue alone does not establish a fix.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:L

Observed

Orleans' activation migration manager failed to migrate a single grain activation on each of two attempts, half a second apart, on one portal pod in the memex-cloud namespace: System.TimeoutException raised inside ActivationMigrationManager.PumpMigrationQueue(SiloAddress targetSilo, Channel<WorkItem> workItems) ({path}:734, per the stack frame). The two failures name different target silos (S10.244.18.146:11111:152699995 and S10.244.6.63:11111:152700320), so in each case the migration work sent to the target silo did not complete within its bound. The stack is entirely Orleans runtime — no MeshWeaver frame appears beyond the silo address.

Where and when

This is the first sighting of this fingerprint on the instance; the two lines are the whole burst — not a steady rate, not multi-pod. They sit inside a wider burst of failures on the same namespace in the same minute, read from the sibling incidents:

  • .NET Thread Pool execution stalls of ~12 s, ~15 s and ~37 s on another memex-cloud pod, the last ending seconds after this window — {path}#5388 ({path}).
  • Grain placement timeouts for messagehub/* grains whose sender silo is S10.244.18.146:11111:152699995 — the exact silo address that was the migration target in this incident's first sample — {path}#5334 ({path}).
  • A SiloUnavailableException naming S10.244.14.1 "is shutting down" on this incident's own pod seconds later — {path}#7002 ({path}).

Impact

Minimal as observed: two single-activation migration failures in a one-off burst on one pod. What a failed migration means for the affected grain — retried, re-queued, or stranded — is not established, and no user-visible symptom is attributed to it.

Not established

  • The cause; the timeout bound that was exceeded is not on the incident.
  • Whether the two migrations eventually completed.
  • Whether this fingerprint recurs outside this window.
  • Whether the correlated sibling incidents share one trigger (a roll, a stall, or a silo leaving the cluster) — the correlation rests on timing and one shared silo identity, nothing more.

Candidate causes and the reading that separates them

  1. Transient cluster-wide stall or overload in that minute. Predicts the target silos were themselves stalled at the time. Reading: each target silo's own LocalSiloHealthMonitor lines (10.244.18.146, 10.244.6.63) for that minute — #5388's stalls were observed on a third pod, not on either target, so this is not yet evidenced for the targets.
  2. Target silos shutting down or evicted mid-migration. Predicts membership or SiloUnavailableException events naming those two silos near the window. Reading: {path} events for those two silo addresses around the burst — #7002's sample names S10.244.14.1, not either target, so this is not yet evidenced either.
  3. Migration racing the departure of the silo being drained. Predicts preceding ActivationMigrationManager activity on the reporting pod naming a source silo with a shutdown reason. Reading: the same log category's informational lines on the reporting pod just before the failures.

Related, not duplicates

No earlier incident carries this fingerprint — first appearance. The sibling tickets above are different log sites and separate defects, but they share the same minute, namespace and (in #5334's case) the silo S10.244.18.146; one underlying cluster event may explain all of them, and whoever picks this up should read the four together.

Where to look

  • Orleans.Runtime.ActivationMigrationManager.PumpMigrationQueue — {path}:734 (path as recorded in the stack frame).
  • Sibling incidents of the same minute: {path} (#5334), {path} (#5388), {path} (#7002).
Evidence
Fingerprint efc4d73fd3e6e697
Category Orleans.Runtime.ActivationMigrationManager
Severity Error
Exception System.TimeoutException
Namespace memex-cloud 2 occurrence(s), 2026-11-03 09:12:07Z → 2026-11-03 09:12:07Z; pods memex-portal-deployment-659bbc4547-w56hx
Occurrences 2
First seen 2026-11-03 09:12:07Z
Last seen 2026-11-03 09:12:07Z
Routing not determined — no configured route matches the category Orleans.Runtime.ActivationMigrationManager. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-11-03 09:12:07Z memex-portal-deployment-659bbc4547-w56hx fail: Orleans.Runtime.ActivationMigrationManager[1491592786]
      Error while migrating 1 grain activations to S10.244.18.146:11111:152699995
      System.TimeoutException: The operation has timed out.
         at Orleans.Runtime.ActivationMigrationManager.PumpMigrationQueue(SiloAddress targetSilo, Channel`1 workItems) in /_/src/Orleans.Runtime/Catalog/ActivationMigrationManager.cs:line 734
2026-11-03 09:12:07Z memex-portal-deployment-659bbc4547-w56hx fail: Orleans.Runtime.ActivationMigrationManager[1491592786]
      Error while migrating 1 grain activations to S10.244.6.63:11111:152700320
      System.TimeoutException: The operation has timed out.
         at Orleans.Runtime.ActivationMigrationManager.PumpMigrationQueue(SiloAddress targetSilo, Channel`1 workItems) in /_/src/Orleans.Runtime/Catalog/ActivationMigrationManager.cs:line 734

Opened automatically from Admin/_LogIncident/efc4d73fd3e6e697. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 2fee425283d46278: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
5h 8m
PR unite (30g)
979

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.