Silo IoPool teardown expired over one in-flight mesh query leaf on memex — Query pool's Observable.Using over MeshQuery+QueryMembers outlived the 30 s join

Start with Threading/IoPool.cs and trace IoPool.Invoke's linked cancellation token, then inspect the Query pool registration and MeshWeaver.Hosting Persistence/Query/MeshQuery's QueryMembers subscription. Compare the leaf's cancellation behavior with PooledLeafObservesItsTokenGuard and read the silo stop timeline for the named pod. Done means establishing why the single Query leaf outlived the 30-second join and identifying a reproducible fix; the incident alone does not establish the cause.

Observed

IoPoolSiloTeardown (the IoPool silo-lifecycle participant at stage First, 30 s bounded join) reported that its join expired over live work during a silo stop, and — unlike the original core issue 2480 occurrences, which named nothing — this one names its subject: Query pool, 1 in-flight leaf, reported as System.Reactive.Linq.ObservableImpl.Using2[[MeshWeaver.Mesh.QueryResultChange1[[MeshWeaver.Mesh.MeshNode, MeshWeaver.Mesh.Contract, v3.1.0.0]], MeshWeaver.Mesh.Contract, v3.1.0.0],[MeshWeaver.Hosting.Persistence.Query.MeshQuery+QueryMembers, MeshWeaver.Hosting, v3.1.0.0]]. One pooled mesh-query leaf (Observable.Using over MeshQuery.QueryMembers producing QueryResultChange<MeshNode>) was still running when the 30 s budget ran out, so the silo released over it. The message's own text attributes this to a leaf that did not observe its cancellation token — that is the framework's diagnosis, not an independently established cause (see below).

This is the same log site and the same defect class as core issue 2480 (18 occurrences of the bare timeout sentence, whose subject was never identified): IoPool.Invoke runs its leaf as return await io(ct) with the subscriber's token linked to the pool-wide token that Drain/Dispose cancel, so a leaf whose returned Task cannot observe ct holds its permit past any drain, the pool's Disposed never completes, and the caller's bounded join expires over it. Related prior findings on this class, neither covering this leaf: Systemorph/MeshWeaver.Feedback#23 (the only measured production site ignoring the token is the embedding HTTP leaf in MeshWeaver.Plugins — the Query pool was NOT implicated there) and Systemorph/MeshWeaver.Feedback#24 (the N leaf(es) survived the join Error branch of this same report is unreachable because IoPool.Disposed carries a hard-coded 0 — which is presumably why the attribution had to take this "Did NOT report" form).

Where and when

Single occurrence, single pod (memex-portal-deployment-78fd5d66b7-lmm7j), tenant memex, at 2026-10-26 22:53:57Z. The log site only fires during a silo stop, so the pod was in its stop sequence at that moment; whether a roll or a single restart put it there was not established from the incident. This is the first occurrence this fingerprint records, and the first at this site since the attribution (Did NOT report: …) made the subject nameable.

Impact

The silo stopped with a pooled mesh-query leaf still running — the stop's bounded join was burned, the pod was released over live work, and whatever that leaf was doing (a live MeshNode change subscription feeding a client) lost its host. One stop, one pod, one leaf: an intermittent shutdown-time defect today, but of the class that made 2480 a three-week hunt when it named nothing, and the one that IoPoolSiloTeardown's own remarks call "the ONLY attribution a subsequent SIGSEGV will get".

Not established

• The cause. Whether the named Query leaf actually ignores the pool token (the 2480 class), or was simply a leaf observing the token whose underlying work took longer than 30 s to unwind (e.g. a slow-to-cancel database command behind QueryMembers), or the cancellation callbacks were starved and never ran at all. The framework's message asserts the first; the evidence here is one timeout line.
• Whether the leaf ever completed after the report, or holds its permit indefinitely (a pool whose Disposed never fires).
• Whether the stop otherwise completed cleanly and whether the process exited shortly after.
• Whether this recurs on every stop of a memex pod or was a one-off.
• Whether other pools reported in the same stop (the sample names only Query).

Candidate causes and the reading that separates them

1. The Query leaf ignores the pool token (the 2480 class, at the persistence-query surface in core). An Observable.Using over MeshQuery.QueryMembers whose subscription work does not observe ct — the same shape as the embedding leaf in Feedback#23, but inside core's own query pipeline, where the core guard PooledLeafObservesItsTokenGuard (added with 2480's fix, scanning core src) should have caught it — whether this leaf shape is genuinely outside what that guard holds, or inside core and missed by it, is itself worth answering. Predicts: the same fingerprint on every stop of a pod with an active mesh query, and a Query pool that never completes. Discriminating reading: the IoPool.Invoke call site that registers the Query pool's leaf — whether the returned Task observes the linked token — plus the next stop of a memex pod in Loki: whether Query=1 appears again.
2. The leaf observes the token, but its underlying work outlived the budget. A QueryMembers read or a PostgreSql command that takes cancellation but unwinds slower than 30 s (cancel of an in-flight database command, Rx teardown queued behind other work). Predicts: the stop completed soon after the report anyway (process exit lines within the following minute), and the fingerprint does not recur on every stop. Discriminating reading: Loki for this pod after 22:53:57Z — the process's exit timing relative to the report.
3. A starved silo: cancellation was requested but the callbacks could not run. The memex starved-silo shape recorded in deploy/helm/values.yaml (2026-07-22 membership-probe starvation) — thread-pool saturation at stop time means the token's registrations never execute. Predicts: the pod's SiloStopTimeline lines (now pinned to Information in Memex.Portal.Distributed/appsettings.json, held by ShippedLoggingLevelsGuard) show large pending work items or GC pause total figures, or a .NET Thread Pool execution stalled warning in the same window. Discriminating reading: that pod's SiloStopTimeline: lines and the thread-pool stall warnings for 22:53–22:55Z.

Where to look

• MeshWeaver.Connection.Orleans IoPoolSiloTeardown / IoPoolRegistry — the report and the per-pool Disposed subscription (TryFinishDisposal publishes the hard-coded 0 per Feedback#24).
• MeshWeaver.Mesh.Contract Threading IoPool.cs — Invoke's return await io(ct) leaf and the token linkage.
• MeshWeaver.Hosting Persistence/Query/MeshQuery — the QueryMembers subscription the report names; the leaf the 30 s join expired over.
• Doc/Architecture/ReadingASiloStop (mesh) — how to line up this pod's SiloStopTimeline / IoPoolSiloTeardown / MeshTeardownHostedService lines against peers' view of the same stop.
• Prior art: core issue 2480; Systemorph/MeshWeaver.Feedback#23 and #24.

Evidence

2026-10-26 22:53:57Z memex-portal-deployment-78fd5d66b7-lmm7j fail: MeshWeaver.Connection.Orleans.IoPoolSiloTeardown[0]
      IoPoolSiloTeardown: pooled I/O did not finish within 00:00:30 — the silo is releasing over live work. A leaf ignored its cancellation token; fix the leaf, do not widen the budget. Did NOT report: Query=1 [System.Reactive.Linq.ObservableImpl.Using`2[[MeshWeaver.Mesh.QueryResultChange`1[[MeshWeaver.Mesh.MeshNode, MeshWeaver.Mesh.Contract, Version=3.1.0.0, Culture=neutral, PublicKeyToken=null]], MeshWeaver.Mesh.Contract, Version=3.1.0.0, Culture=neutral, PublicKeyToken=null],[MeshWeaver.Hosting.Persistence.Query.MeshQuery+QueryMembers, MeshWeaver.Hosting, Version=3.1.0.0, Culture=neutral, PublicKeyToken=null]]].

This log site's previous ticket, https://github.com/Systemorph/MeshWeaver/issues/7049, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

_{Opened automatically from Admin/_LogIncident/6e502415adcf0ae7. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 4806d28cbbe5e043: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}