Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Terminal NotFound NACK undeliverable to a portal hub whose pod-hub registration was already released

Open
  • Is something wrong?
#7,001 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
backend, distributed-systems

Research direction

Read the post-failure NACK path in MeshWeaver.Hosting.Orleans.RoutingGrain, then inspect PodHubGrain.Deliver at src/MeshWeaver.Hosting.Orleans/PodHubGrain.cs:187 and the Detach release tombstone. For the stranded-claim possibility, follow OrleansRoutingService.RegisterStream disposal and the claim lifecycle described in Pod-Hub Claim Re-assertion. Determine whether the unreachable-reply residual needs handling and whether this incident is a sibling of #5703.

Written by the indexing model from the issue text.

Description

bug sev:L

Observed

RoutingGrain resolved delivery xa8UiEuTHU6Rf0RNpCqO-Q for the node Event/UzhEmbaGastvortrag2026/emba-solution-and-notes-bob to a terminal NotFound verdict — the node does not exist; the closest ancestor is Event/UzhEmbaGastvortrag2026 (remainder emba-solution-and-notes-bob). Reporting that verdict back to the sender hub portal/f6hQZWqMlXms8zUCSVW4fs_NWrhPerrd8-eEF2ntBdI failed on both legs: the directed pod-hub call was answered with MeshWeaver.Connection.Orleans.PodHubNotHereException { Released = true } from PodHubGrain.Deliver (src/MeshWeaver.Hosting.Orleans/PodHubGrain.cs:187) — the hub's registration was disposed and the address tombstoned — and the stream fallback found no live subscriber either, so the terminal NACK landed nowhere and was logged as undeliverable.

Both verdict machines behaved as designed: the NotFound for a genuinely absent node, and the released-tombstone terminal verdict (Pod-Hub Delivery Roll Plan — a released address answers terminally). What remains is the irreducible last row of that page's residual table — when a reply cannot be delivered, the party that needs to know is precisely the unreachable one.

Where and when

Single occurrence on one pod in the memex-cloud namespace. The sender is a portal/… per-circuit hub address, so the scope is one client circuit, not a fleet-wide condition. The line reports 0 earlier undeliverable NACKs since the last such line — a one-shot event, not a persistent refusal window like #5703's 18.

Impact

If the requesting circuit had closed, the requester is gone and the undeliverable NACK is log noise with no user impact. If the circuit was alive, that one request for the missing node burned its full read budget with no error surfaced to the client.

Not established

  • Whether the requesting circuit was alive at the moment the NACK was undeliverable. The tombstone says the owner disposed its registration; whether that was a normal circuit close, a pod restart, or a stranded claim is not on the evidence.
  • Whether the fingerprint recurs — one occurrence is not a trend.
  • Whether the target node Event/UzhEmbaGastvortrag2026/emba-solution-and-notes-bob was deleted or never existed; no version history of it was read.
  • The framework source itself was not readable from this mesh; the component and mechanism names come from the incident's stack and the architecture pages (Pod-Hub Delivery Roll Plan, Dead-Circuit Fan-Out Storm, The Pod-Hub Claim Must Be Re-Asserted).

Candidate causes and the reading that separates them

  1. Normal circuit teardown racing the reply — the client closed while the request was in flight; the owner-side RegisterStream disposal detached and tombstoned the hub, so the reply arrived at a corpse. Predicts exactly this shape: a single undeliverable NACK at teardown and no further activity from that hub address. Reading that rules it in: no deliveries, heartbeats or requests from portal/f6hQ… at or after the incident timestamp on any pod.
  2. Stranded claim (the #2938/#2915 shape, client variant) — the circuit was alive but its pod-hub claim lost its mapping (Pod-Hub Claim Re-assertion): an Orleans client asserts its claim once, and a grain-directory re-partition strands it permanently, so every reply and NACK to it is refused for the life of the connection. Predicts repeat occurrences and per-request budget timeouts for that client. Reading that rules it in: further occurrences of this fingerprint for the same hub address, or evidence the client was issuing requests at the time.

Cause not established; the single-occurrence evidence does not discriminate between the arms. The teardown arm is consistent with everything on the incident; the stranded-claim arm would carry user impact.

Related

#5703 covers the same residual at the same log site with a different failure leg ("no pod-hub activation claims that address AND its stream has no live subscriber" — no tombstone involved); this incident is the Released = true arm of the same "reply to an unreachable requester" residual. Whoever picks this up may close it as a sibling of #5703.

Where to look

  • MeshWeaver.Hosting.Orleans.RoutingGrain — the post-failure NACK path behind the undeliverable-NACK log site (the PostFailure leg, whose stream fallback landed per #2357).
  • MeshWeaver.Hosting.Orleans.PodHubGrain.Deliver (src/MeshWeaver.Hosting.Orleans/PodHubGrain.cs:187) and Detach's release tombstone — the verdict side, working as designed here.
  • OrleansRoutingService.RegisterStream disposal and the pod-hub claim lifecycle — for the stranded-claim arm, see Pod-Hub Claim Re-assertion.
Evidence
Fingerprint 72db079a7c086f6f
Category MeshWeaver.Hosting.Orleans.RoutingGrain
Severity Error
Exception MeshWeaver.Connection.Orleans.PodHubNotHereException
Top frame MeshWeaver.Hosting.Orleans.PodHubGrain.Deliver(IMessageDelivery delivery)
Namespace memex-cloud
Pods memex-portal-deployment-5fd6d747d-xbct6
Occurrences 1
First seen 2026-10-20 13:09:41Z
Last seen 2026-10-20 13:09:41Z
Routing not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-20 13:09:41Z memex-portal-deployment-5fd6d747d-xbct6 fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Cannot deliver the NotFound DeliveryFailure for xa8UiEuTHU6Rf0RNpCqO-Q to sender portal/f6hQZWqMlXms8zUCSVW4fs_NWrhPerrd8-eEF2ntBdI: no available transport carried it (see the exception for which leg failed). The sender will wait out its own request budget — the original failure was: No node found at 'Event/UzhEmbaGastvortrag2026/emba-solution-and-notes-bob'. Closest ancestor is 'Event/UzhEmbaGastvortrag2026' (remainder='emba-solution-and-notes-bob').. 0 earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.
      MeshWeaver.Connection.Orleans.PodHubNotHereException: 'portal/f6hQZWqMlXms8zUCSVW4fs_NWrhPerrd8-eEF2ntBdI' was RELEASED by its owner: the registration that claimed it was disposed, so no silo will serve it again through the pod-hub transport. This is a terminal verdict, not a roll window.
         at MeshWeaver.Hosting.Orleans.PodHubGrain.Deliver(IMessageDelivery delivery) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.Orleans/PodHubGrain.cs:line 187
         at Orleans.Runtime.TaskRequest`1.Invoke() in /_/src/Orleans.Core.Abstractions/Runtime/GrainReference.cs:line 739
      --- End of stack trace from previous location ---
         at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Core/Core/GrainMethodInvoker.cs:line 132
         at MeshWeaver.Hosting.Orleans.AccessContextGrainCallFilter.Invoke(IIncomingGrainCallContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.Orleans/AccessContextGrainCallFilter.cs:line 38
         at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Core/Core/GrainMethodInvoker.cs:line 94
         at Orleans.Runtime.InsideRuntimeClient.Invoke(IGrainContext target, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 319
         at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Seria…[truncated]

Opened automatically from Admin/_LogIncident/72db079a7c086f6f. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 168d0f4e80a7fec1: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
5h 57m
Merged PRs (30d)
967

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.