MeshNodeStreamCache hands a new subscriber the disposed $security-access:ClaimsDeepfield query entry, failing the Overview render on ClaimsDeepfield

First check pod lifecycle and roll or restart records for pod memex-portal-deployment-68b656cd88-thr49 around 2026-10-31T08:14Z; this is the cheapest way to distinguish shutdown from a live-pod race. Then inspect MeshNodeStreamCache query-entry lifecycle (GetQuery, EvictFaultedQuery, Dispose) and the pod's teardown logs, alongside LayoutAreaHost and PermissionEvaluator.ObserveEffectiveAssignments. Done means establishing why the disposed entry was returned instead of a fresh one and addressing that path; the cause is not established in the report.

Observed

A render of layout area Overview on node ClaimsDeepfield failed on the portal pod memex-portal-deployment-68b656cd88-thr49 (namespace memex-cloud) with System.ObjectDisposedException:

MeshNodeStreamCache query '$security-access:ClaimsDeepfield' has released the shared connection this subscription would ride: the owner is disposed, its upstream is unsubscribed, and a subscriber arriving now would otherwise wait on a replay nothing will feed.

$security-access:ClaimsDeepfield is the partition-wide grant read that feeds the permission fold (PermissionEvaluator.ObserveEffectiveAssignments → IMeshNodeStreamCache.GetQuery, one process-wide synced-query entry per partition; see Access Control Architecture and The MeshNode Stream Cache). The exception is the cache's deliberate terminal from the #5011 family — it exists so a subscriber arriving on a torn-down entry faults loudly rather than waiting forever on a Replay(1) nothing will feed. What was not supposed to happen is that the subscriber reached the disposed entry at all: a new read should resolve to a fresh entry, not the corpse.

Where and when

One occurrence, one pod, one instant (2026-10-31T08:14:17Z); first and last sighting are the same timestamp, so this is a single race window, not a burst or a steady rate. No deploy window, roll or restart has been checked against that instant — see "Not established".

Impact

The Overview area of ClaimsDeepfield failed to render for at least one viewer at that instant. Because the faulted entry is the partition's shared grant read, every permission check on the ClaimsDeepfield partition rides the same handle; a torn-down entry that keeps admitting (and faulting) new subscribers would degrade every gated read in that partition, not just this render. Only one sighting exists, so no broader degradation is evidenced — the guard prevented the silent-stall alternative, which would have parked the render indefinitely.

Not established

• The cause. The evidence fixes the site (a new subscriber on $security-access:ClaimsDeepfield met a released entry) but not which reaper released it, nor why the subscriber was routed to the disposed entry rather than a fresh one.
• Whether the pod was leaving. The documented ObjectDisposedException-at-disposal class (#5958, #6078) fires on pods inside a roll or restart; LayoutAreaHost's render, unlike MeshNodeEditor, has no IsLeaving() classification recorded here, and no pod-lifecycle correlation has been read for 08:14Z.
• Whether it recurs or self-heals. A retry of the page render should re-create the entry (the cache's GetQuery/GetStreamRaw re-probe path); no second occurrence confirms or refutes that.
• Whether other consumers saw the same terminal — e.g. a [RequiresPermission] delivery gate or RlsNodeValidator fold touching ClaimsDeepfield in the same seconds — which would say the entry kept faulting new subscribers rather than faulting one.
• Whether the entry carried a prior fault (storage error, MeshNodeContentDegradedException, or a transient-breaker record for the ClaimsDeepfield partition) whose eviction raced this subscription.

Candidate causes and the reading that separates them

1. A live-pod eviction raced a new subscriber. One of the four faulted-entry teardown triggers ({path} breaker expiry, GetStreamRaw's faulted-entry guard, the change-feed ResetFailureState) or the query-side idle release tore the $security-access:ClaimsDeepfield entry down while LayoutAreaHost's data read was attaching; the detach→claim→teardown protocol handed the arriving subscriber the disposal terminal instead of a fresh entry. Predicts: a {path} trace on that pod in the same second, and a retry that succeeds. Reading: the cache's teardown logs ({path} lines) on …-thr49 at 08:14:17Z, plus a re-open of the page.
2. The process was leaving. The cache (or cache hub) was disposed as part of a pod {path}, and the render's read arrived during host shutdown — the #5958/#6078 shape, with no consumer-side IsLeaving() classification on the layout path. Predicts: an {path} roll or restart covering 08:14Z for that pod, and sibling disposal terminals from other long-lived readers in the same window. Reading: {path} on the control instance and the pod's {path} times around 2026-10-31T08:14Z.
3. A prior fault on the entry set up the race. The ClaimsDeepfield grant read faulted earlier (storage, degradation, or a partition event) and the faulted-entry eviction ran while the layout render's subscription arrived — the eviction reaching the reader before the reader could attach to the replacement entry. Predicts: an error or breaker line for $security-access:ClaimsDeepfield (or ClaimsDeepfield storage) minutes before 08:14Z on that pod. Reading: Loki error lines for the ClaimsDeepfield partition and the $security-access query id on that pod in the preceding minutes.

Cause not established; all three are consistent with the single sighting. The first reading worth taking is the pod-lifecycle check (it is the cheapest and it splits cause 2 from causes 1 and 3).

Where to look

• Implementation: {path} — the query-side entry lifecycle (GetQuery/EvictFaultedQuery, the detach→claim→teardown protocol, Dispose), and the guard whose message text this is; contract {path} (IMeshNodeStreamCache).
• The consumer that surfaced it: MeshWeaver.Layout.Composition.LayoutAreaHost (Rendering failed for area …, FailRendering).
• The fold leg the entry serves: PermissionEvaluator.ObserveEffectiveAssignments / SecurityQueries — Access Control Architecture → "Two reads per path".
• The disposal-terminal family this guard belongs to: The MeshNode Stream Cache → "A forced teardown ENDS its readers" (#5011), "A long-lived editor on a leaving process" (#5958), "A hub-owned read ends with its hub" (#6078); and Disposed Scopes and Dying Hubs for the discriminator set around ObjectDisposedException.

No existing incident or issue covering this component and symptom was found in the mesh; no duplicate is known.

Severity note: judged sev:M — one occurrence, one pod, one area render, with the guard having prevented the worse (silent indefinite stall) outcome. If it recurs, this is an intermittent user-visible render failure on a shared permission-fold input and should be re-ruled upward.

Evidence

2026-10-31 08:14:17Z memex-portal-deployment-68b656cd88-thr49 fail: MeshWeaver.Layout.Composition.LayoutAreaHost[0]
      Rendering failed for area Overview on ClaimsDeepfield
      System.ObjectDisposedException: MeshNodeStreamCache query '$security-access:ClaimsDeepfield' has released the shared connection this subscription would ride: the owner is disposed, its upstream is unsubscribed, and a subscriber arriving now would otherwise wait on a replay nothing will feed.
      Object name: 'MeshNodeStreamCache query '$security-access:ClaimsDeepfield''.

_{Opened automatically from Admin/_LogIncident/afed65253247024b. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site f72c2865c1d159ff: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}