Orleans ActivationMigrationManager times out migrating single grain activations to two target silos on memex-cloud

Open
  • Is something wrong?
#7,861 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
distributed-systems

Research direction

Start with Orleans.Runtime.ActivationMigrationManager.PumpMigrationQueue at ActivationMigrationManager.cs:734, then read its informational log lines just before the failures. Compare the reporting pod's migration activity with the target silos' LocalSiloHealthMonitor and membership logs, and read incidents #5334, #5388, and #7002 for the same minute. Done means identifying a supported cause or documenting what evidence is still missing; this issue alone does not establish a fix.

Written by the indexing model from the issue text.

Description

bug sev:L

Observed

Orleans' activation migration manager failed to migrate a single grain activation on each of two attempts, half a second apart, on one portal pod in the memex-cloud namespace: System.TimeoutException raised inside ActivationMigrationManager.PumpMigrationQueue(SiloAddress targetSilo, Channel<WorkItem> workItems) ({path}:734, per the stack frame). The two failures name different target silos (S10.244.18.146:11111:152699995 and S10.244.6.63:11111:152700320), so in each case the migration work sent to the target silo did not complete within its bound. The stack is entirely Orleans runtime — no MeshWeaver frame appears beyond the silo address.

Where and when

This is the first sighting of this fingerprint on the instance; the two lines are the whole burst — not a steady rate, not multi-pod. They sit inside a wider burst of failures on the same namespace in the same minute, read from the sibling incidents:

  • .NET Thread Pool execution stalls of ~12 s, ~15 s and ~37 s on another memex-cloud pod, the last ending seconds after this window — {path}#5388 ({path}).
  • Grain placement timeouts for messagehub/* grains whose sender silo is S10.244.18.146:11111:152699995 — the exact silo address that was the migration target in this incident's first sample — {path}#5334 ({path}).
  • A SiloUnavailableException naming S10.244.14.1 "is shutting down" on this incident's own pod seconds later — {path}#7002 ({path}).

Impact

Minimal as observed: two single-activation migration failures in a one-off burst on one pod. What a failed migration means for the affected grain — retried, re-queued, or stranded — is not established, and no user-visible symptom is attributed to it.

Not established

  • The cause; the timeout bound that was exceeded is not on the incident.
  • Whether the two migrations eventually completed.
  • Whether this fingerprint recurs outside this window.
  • Whether the correlated sibling incidents share one trigger (a roll, a stall, or a silo leaving the cluster) — the correlation rests on timing and one shared silo identity, nothing more.

Candidate causes and the reading that separates them

  1. Transient cluster-wide stall or overload in that minute. Predicts the target silos were themselves stalled at the time. Reading: each target silo's own LocalSiloHealthMonitor lines (10.244.18.146, 10.244.6.63) for that minute — #5388's stalls were observed on a third pod, not on either target, so this is not yet evidenced for the targets.
  2. Target silos shutting down or evicted mid-migration. Predicts membership or SiloUnavailableException events naming those two silos near the window. Reading: {path} events for those two silo addresses around the burst — #7002's sample names S10.244.14.1, not either target, so this is not yet evidenced either.
  3. Migration racing the departure of the silo being drained. Predicts preceding ActivationMigrationManager activity on the reporting pod naming a source silo with a shutdown reason. Reading: the same log category's informational lines on the reporting pod just before the failures.

Related, not duplicates

No earlier incident carries this fingerprint — first appearance. The sibling tickets above are different log sites and separate defects, but they share the same minute, namespace and (in #5334's case) the silo S10.244.18.146; one underlying cluster event may explain all of them, and whoever picks this up should read the four together.

Where to look

  • Orleans.Runtime.ActivationMigrationManager.PumpMigrationQueue — {path}:734 (path as recorded in the stack frame).
  • Sibling incidents of the same minute: {path} (#5334), {path} (#5388), {path} (#7002).
Evidence
Fingerprint efc4d73fd3e6e697
Category Orleans.Runtime.ActivationMigrationManager
Severity Error
Exception System.TimeoutException
Namespace memex-cloud 2 occurrence(s), 2026-11-03 09:12:07Z → 2026-11-03 09:12:07Z; pods memex-portal-deployment-659bbc4547-w56hx
Occurrences 2
First seen 2026-11-03 09:12:07Z
Last seen 2026-11-03 09:12:07Z
Routing not determined — no configured route matches the category Orleans.Runtime.ActivationMigrationManager. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-11-03 09:12:07Z memex-portal-deployment-659bbc4547-w56hx fail: Orleans.Runtime.ActivationMigrationManager[1491592786]
      Error while migrating 1 grain activations to S10.244.18.146:11111:152699995
      System.TimeoutException: The operation has timed out.
         at Orleans.Runtime.ActivationMigrationManager.PumpMigrationQueue(SiloAddress targetSilo, Channel`1 workItems) in /_/src/Orleans.Runtime/Catalog/ActivationMigrationManager.cs:line 734
2026-11-03 09:12:07Z memex-portal-deployment-659bbc4547-w56hx fail: Orleans.Runtime.ActivationMigrationManager[1491592786]
      Error while migrating 1 grain activations to S10.244.6.63:11111:152700320
      System.TimeoutException: The operation has timed out.
         at Orleans.Runtime.ActivationMigrationManager.PumpMigrationQueue(SiloAddress targetSilo, Channel`1 workItems) in /_/src/Orleans.Runtime/Catalog/ActivationMigrationManager.cs:line 734

Opened automatically from Admin/_LogIncident/efc4d73fd3e6e697. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 2fee425283d46278: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
5h 10m
Merged PRs (30d)
979

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.