Sync-stream hubs still at RunLevel=Starting hold never-finishing DeliveryFailure turns through mesh teardown, overrunning DrainAsync's 30 s bound

Read MeshTeardownExtensions.DrainAsync and MeshTeardownHostedService, then trace the delivery pipeline's DeliveryFailure handling and the sync-stream hub handlers. The report does not identify the handler's cause or the failing delivery; inspect the two hubs' failure payloads and Loki lines from 07:44–07:58Z, then check for disposal wedge or discard events. Done means identifying why the turns hang and whether teardown cancellation reaches them.

Observed

During pod shutdown, MeshWeaver.Mesh.MeshTeardownExtensions.DrainAsync (reported by MeshWeaver.Hosting.MeshTeardownHostedService, 30 s TeardownTimeout) gave up on mesh {path}: the mesh root and its {path} child sit at RunLevel=DisposeHostedHubs, Disposal=Pending with all queues empty, held by two hosted sync hubs — {path} and {path} — which are still at RunLevel=Starting (openGates=1) and whose action blocks have each been executing a single DeliveryFailure turn for 501 240 ms (~8.4 min) and 801 280 ms (~13.4 min), with the pumps draining (drainsInFlight=1) and one further turn queued (buffer=1) behind each. Both wedged turns began minutes before the teardown started (≈07:49:33Z and ≈07:44:33Z against the 07:57:54Z log line).

Two structural facts about the wedged hubs, read against {path}:

• The executing turn is not the hub's InitializeHubRequest, so the bring-up cancel path ("a hub that never finished starting") does not reach it.
• Each hub's ShutdownRequest queues FIFO behind the wedged turn, so neither hub ever begins disposing — the stall detector ([DISPOSE-WEDGE], CancelExecution()) arms on the hub's own Dispose and therefore never fires. No cooperative cancellation path reached these turns at all; the teardown's outer bound reported, as designed, and the host proceeded over unfinished work.

Where and when

One occurrence, one pod (memex-portal-deployment-5897bcf6d4-hdssc), namespace memex-cloud, at a single instant during pod termination. The wedged turns predate the shutdown by 8–13 minutes; the timeout itself is the designed caller bound reporting, not a crash or a burst.

Impact

Judged from the numbers: on this burst the overrun is bounded — the host logs, proceeds to HostOptions.ShutdownTimeout and the pod grace ceiling ends the process; no serving degradation is evidenced on this incident. The concern is the wedged-turn shape itself: a DeliveryFailure turn that does not finish for 8–13 minutes on a hub still mid-bring-up, reachable by no cancellation path in the teardown design.

Related report — possible same family, not established

{path} (2026-10-28, one day earlier, same memex-cloud instance): a serving cache hub {path} wedged executing a single DeliveryFailure for ~2.5 h, queue growing to ~17k, ~40% of MCP {path} timing out. Same message shape (one never-finishing DeliveryFailure turn), same instance, different hub role and consequence. Not established that the two are the same defect; if confirmed, this incident's severity should be re-ruled upward (the sibling report describes instance-wide degradation).

Not established

• The cause of the turns not finishing — no handler stack, await site or DeliveryFailure payload was read.
• What delivery each DeliveryFailure reports (the failing delivery's target and exception are inside the message, not in the log line).
• Whether it still occurs, whether other pods or log sites show the same wedged turn, and whether this pod was part of a roll (only that MeshTeardownHostedService ran, i.e. SIGTERM, is evidenced).
• Which {path} the sync/* hubs belong to (ids are instance-scoped; the per-instance identity is masked).
• Whether [DISPOSE-WEDGE]/DISPOSAL DEADLOCK lines exist for these two hub addresses (see candidate cause 2).

Candidate causes and the reading that separates them

1. The DeliveryFailure handler on a sync hub blocks on an await that never completes — the same shape the 2026-10-28 report describes on a serving cache hub. Predicts: a dump of the executing turn parks on a named await, and the DeliveryFailure payload names the delivery and target that failed. Reading: the DeliveryFailure content for both hub ids (its inner failure names the failed delivery and address), and the Loki lines for pod memex-portal-deployment-5897bcf6d4-hdssc covering 07:44–07:58Z, which show what delivery failed and what the hub was doing when the turn was accepted.
2. A structural gap: no cancellation path reaches a wedged turn on a hub that never began disposing (mechanism for why 8–13 min elapsed untouched, not the cause of the turn not finishing). Predicts: no [DISPOSE-WEDGE] (7311) / DISPOSAL DEADLOCK DETECTED (7312) lines and no [DISPOSE-DISCARD] naming these two hub addresses in this window — their presence would move the finding to a turn that received and ignored cancellation; their absence confirms the gap. Reading: Loki search for those event ids against the two sync addresses.
3. The DeliveryFailure is a downstream symptom of the never-finished bring-up (RunLevel=Starting, openGates=1): a delivery failed against the half-started hub and its failure handler interacts with the still-shut initialization gate. Predicts: the DeliveryFailure's inner exception names the bring-up or the gate rather than an external target. Reading: the same payload as (1) separates this from (1) by the ordering of the inner failure.

Where to look

• MeshWeaver.Mesh.MeshTeardownExtensions.DrainAsync and MeshWeaver.Hosting.MeshTeardownHostedService (the 30 s bound that reported) — meshweaver partition.
• The DeliveryFailure handling path in the delivery pipeline (a [CanBeIgnored] message per the teardown docs) and the sync/* (sync-stream) hubs' handlers.
• {path} (the two layers, the bring-up exception, the stall verdicts), {path}, {path}, {path}.
• Related report: {path}.

Severity note: sev:M because this burst alone is a bounded shutdown overrun on one pod with no evidenced serving impact; the sibling report from the previous day, if confirmed as the same wedged-DeliveryFailure defect, warrants a human re-ruling upward.

Evidence

2026-10-29 07:57:54Z memex-portal-deployment-5897bcf6d4-hdssc fail: MeshWeaver.Hosting.MeshTeardownHostedService[0]
      MeshTeardownHostedService: mesh {path} did not finish draining within 00:00:30 — the host proceeds to dispose its container over work that has not finished. Find what is still going down; do not widen the budget.
      Hub {path} RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False,drainsAwaitingScheduler=0) Registrants=35
      HostedHubs (1):
        Hub {path} RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False,drainsAwaitingScheduler=0) Registrants=5
        HostedHubs (2):
          Hub {path} RunLevel=Starting Disposal=Pending Queue(buffer=1,deferred=0,drainsInFlight=1,openGates=1,draining=True,drainsAwaitingScheduler=0) Executing(DeliveryFailure, 501240ms) Registrants=6
          Hub {path} RunLevel=Starting Disposal=Pending Queue(buffer=1,deferred=0,drainsInFlight=1,openGates=1,draining=True,drainsAwaitingScheduler=0) Executing(DeliveryFailure, 801280ms) Registrants=6
      
      System.TimeoutException: Mesh {path}: disposal did not complete within 00:00:30. Something on the action block is not finishing — find it; do not widen the budget.
      Hub {path} RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False,drainsAwaitingScheduler=0) Registrants=35
      HostedHubs (1):
        Hub {path} RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False,drainsAwaitingScheduler=0) Registrants=5
        HostedHubs (2):
          Hub {path} RunLevel=Starting Disposal=Pending Queue(buffer=1,deferred=0,drainsInFlight=1,openGates=1,draining=True,drainsAwaitingScheduler=0) Executing(Deliv…[truncated]

_{Opened automatically from Admin/_LogIncident/b90bf68dc7209342. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 950626c18f056bf0: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}