RoutingGrain cannot return a terminal NotFound NACK to a sender hub addressed by a deleted node's path

Start in MeshWeaver.Hosting.Orleans.RoutingGrain: find the undeliverable-NACK log site (the PostFailure leg) and the sender-resolution branch that emits "neither a pod-process hub nor a node". Then trace the per-node hub lifecycle described in Doc/Architecture/PerProcessInfrastructureHubs and the residual table in Doc/Architecture/PodHubDeliveryRollPlan. First diagnostic step is checking delivery timestamps of the two deliveries against the space's lastModified to separate the two candidate causes. Root cause is not established and this touches Orleans routing internals — not a first-timer issue.

Observed

RoutingGrain could not deliver the terminal NotFound verdict for deliveries AwhBDb7ZI0uQs7JXW6cn7A and D54Kyl04ZE-Ho36tbxPjVw back to their sender — a hub addressed by the node path BusinessPlan2026/BusinessCaseBrief. Both legs of the report failed:

• the directed pod-hub leg refused terminally: the pod-process hub portal/KOlMVZkYYBG7eoPV6_IWHn78zV_yvjgqSwAaFVzQh3k that claimed the sender was RELEASED by its owner (the registration was disposed; the tombstone that answered is on silo 10.244.19.185:11111@151918968) — a closed circuit;
• the sender-resolution fallback found the sender's host address BusinessPlan2026/BusinessCaseBrief is neither a pod-process hub nor a node (closest node BusinessPlan2026, remainder BusinessCaseBrief), so no directed transport reaches it either.

The terminal NACK therefore landed nowhere, and the sender — whoever it was — waits out its own request budget instead of learning the verdict.

Both verdict machines behaved as designed: the release tombstone per Dead-Circuit Fan-Out Storm, and the "reply's own requester" residual, the last row of the residual table in Pod-Hub Delivery — the Transport Swap and its Roll Plan. The new shape here is the second leg: the sender address is a node path, and the node behind it no longer exists.

Where and when

Two occurrences 1.8 s apart on a single pod in the memex namespace, same sender both times — a one-shot event, not a refusal window (the line reports 0 earlier undeliverable NACKs). The sender address is a per-node hub address, so the scope is one client circuit on one node's view, not a fleet condition.

Correlation read on this mesh: the node BusinessPlan2026/BusinessCaseBrief exists nowhere in the mesh and is not in the BusinessPlan2026 space's index; the same missing node produced MarkdownEditorView value-binding failures in area Comments/CreateForm/Editor on the same pod 17 minutes before this burst (filed as #6079), and the BusinessPlan2026 space was last modified by rbuergi at 2026-10-25T08:27:45Z — between the editor failures (08:21) and this burst (08:38). The burst is consistent with the tail of that deletion: a view bound to the deleted node kept a hub registered under its path, the circuit closed, and two in-flight deliveries for the deleted node could not report their verdict back.

Impact

The pod-hub claim was already released when the NACK was undeliverable, so on this evidence the lost verdicts cost nothing user-visible: the requester's circuit is gone and the budget it waits out is a corpse's. The wedge shape — a live sender burning its full request budget on a verdict it never receives — is the same residual as #7001, but nothing in this burst shows a live sender.

Not established

• Whether the two deliveries were issued while the circuit was still open (in flight across teardown) or after the claim was already released — the evidence shows the claim released at NACK time, not at request time.
• Whether BusinessPlan2026/BusinessCaseBrief was deleted around 08:27 or had never existed under that id — the space has no readable version history from this mesh; the timing correlation is circumstantial.
• Whether this fingerprint recurs; two occurrences on one pod is not a trend.
• Whether other log sites show the same "neither a pod-process hub nor a node" sender-resolution failure — only this site was examined.
• The framework source is not readable from this mesh; the component and mechanism names come from the incident's own text and the architecture pages.

Candidate causes and the reading that separates them

1. Circuit teardown racing in-flight replies (the #7001 shape, node-address variant) — the client closed while two reads for the deleted node were in flight; the pod-hub claim was released, the tombstone answered, and the sender-resolution fallback found only a deleted node's path. Predicts no further traffic from sender BusinessPlan2026/BusinessCaseBrief after the burst. Separating reading: deliveries, heartbeats or requests from that address on any pod at or after 2026-10-25T08:38:17Z.
2. A per-node hub outliving its node — the hub registered under the node's path kept serving a saved view after the node's deletion, and the requests were issued against an already-deleted target; the circuit then closed before the verdicts returned. Predicts the editor's NotFound binding failures recurring on the same pod (as #6079 shows they did at 08:14 and 08:21) and a repeat of this fingerprint for every saved view addressing a deleted node. Separating reading: the original request timestamps of deliveries AwhBDb7ZI0uQs7JXW6cn7A and D54Kyl04ZE-Ho36tbxPjVw relative to the space's lastModified (08:27:45Z).

Cause not established; both arms fit, and they differ only in whether the requests pre- or post-date the deletion — which the delivery timestamps would settle.

Related

• #7001 — the same undeliverable-NACK residual at this log site, Released = true arm with a portal/… circuit sender (filed 2026-10-25). This incident is the node-addressed arm of the same residual, not a duplicate: the second failure leg differs (sender resolution, not the stream fallback).
• #6079 — the MarkdownEditorView binding failure for the same missing node BusinessPlan2026/BusinessCaseBrief on the same pod, minutes earlier; the view-layer symptom of the deletion. #6074 enumerates the wider deleted-node view fan-out family.

Where to look

• MeshWeaver.Hosting.Orleans.RoutingGrain — the undeliverable-NACK log site (the PostFailure leg) and the sender-resolution branch that produces "neither a pod-process hub nor a node".
• The per-node hub lifecycle — which component registers a hub under a node path such as BusinessPlan2026/BusinessCaseBrief, and how that registration's lifetime relates to the node's existence (Per-Process Infrastructure Hubs, Dead-Circuit Fan-Out Storm, Pod-Hub Delivery Roll Plan residual table).
• The delete history of BusinessCaseBrief under the BusinessPlan2026 space (no readable version history from this mesh).

Evidence

2026-10-25 08:38:15Z memex-portal-deployment-964d9db5c-59bzj fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Cannot deliver the NotFound DeliveryFailure for AwhBDb7ZI0uQs7JXW6cn7A to sender BusinessPlan2026/BusinessCaseBrief: no available transport carried it (see the exception for which leg failed). The sender will wait out its own request budget — the original failure was: Directed delivery to pod hub 'portal/KOlMVZkYYBG7eoPV6_IWHn78zV_yvjgqSwAaFVzQh3k' was refused: its OWNER RELEASED the address (the registration that claimed it was disposed — a closed circuit, a torn-down hub). Nobody will claim it again, so this is a TERMINAL verdict: the sender's owner-side client-subscription eviction drops the server-side stream it was still fanning out to, instead of pushing to the corpse until idle release. The tombstone that answered is on silo '10.244.19.185:11111@151918968'.. 0 earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.
      System.InvalidOperationException: The sender's host address 'BusinessPlan2026/BusinessCaseBrief' is neither a pod-process hub nor a node (closest node 'BusinessPlan2026', remainder 'BusinessCaseBrief'), so no directed transport reaches it — forward traffic to it is answered NotFound too.
2026-10-25 08:38:17Z memex-portal-deployment-964d9db5c-59bzj fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Cannot deliver the NotFound DeliveryFailure for D54Kyl04ZE-Ho36tbxPjVw to sender BusinessPlan2026/BusinessCaseBrief: no available transport carried it (see the exception for which leg failed). The sender will wait out its own request budget — the original failure was: Directed delivery to pod hub 'portal/KOlMVZkYYBG7eoPV6_IWHn78zV_yvjgqSwAaFVzQh3k' was refused: its OWNER RELEASED the address (the registration that claimed it was disposed — a closed circuit, a torn-down hub). Nobody will claim it again, so this is a TERMINAL verdict: the sender's owner-side client-subscription eviction drops the server-side stream it was still fanning out to, instead of pushing to the corpse until idle release. The tombstone that answered is on silo '10.244.19.185:11111@151918968'.. 0 earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.
      System.InvalidOperationException: The sender's host address 'BusinessPlan2026/BusinessCaseBrief' is neither a pod-process hub nor a node (closest node 'BusinessPlan2026', remainder 'BusinessCaseBrief'), so no directed transport reaches it — forward traffic to it is answered NotFound too.

_{Opened automatically from Admin/_LogIncident/d4f72b23048c71a9. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site d65680c80925c97c: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}