Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Triage sweep run report lost: ledger update on Hosting/Triage/Status times out waiting for node hydration

Closed
  • Is something wrong?
#5,765 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
observability

Research direction

Start at TriageIssueSweep and trace the AddTriageIssueSweep schedule to the RecordOrFault write to Hosting/Triage/Status; then inspect MeshWeaver.Mesh.MeshNodeStreamHandle and the per-node hub activation logs. If the issue recurs, check whether the owner hydrates the node and whether RLS affects the system-security identity. Done means identifying why the initial state fails to arrive and ensuring the run is recorded; the issue reports one occurrence and no established cause.

Written by the indexing model from the issue text.

Description

bug sev:L

What is failing

The periodic Triage Issue Sweep completes its run (Ok, 144 unclassified found) but cannot record the run on the Hosting/Triage/Status ledger node: the mesh update stream for that node raises a TimeoutException (inner: MeshNodeStreamHandle.BaseStateTimeoutException — no change item reached the mirror within 30s, no snapshot ever hydrated). The run's counters are therefore lost from the Status page until the next run records.

Probable cause

Most likely (moderate confidence) a transient per-node hub activation/hydration stall for Hosting/Triage/Status: the node exists, is active and up to date (version 70, last written ~10 minutes before the failure), which rules out a misspelled path or deleted node. The failure occurred inside a window in which the same fleet reported SiloUnavailableException bursts on the memory queues and a FleetWatch StuckRoll alarm — the portal pod's mesh fabric was unstable, and an owner that ACKed the subscribe but produced no initial frame reads identically to this timeout (the exception's own text calls out that case). Less likely: RLS silently rejecting the write, or the hub's MeshDataSource failing to load the node from persistence — neither is corroborated, since the node was writable before and after.

Impact

Low. A single occurrence, on a single pod, in a background monitor: one sweep run's counters (sweepRuns, sweepFound, lastSweep*) are missing from the Status page for that hour. The sweep itself succeeded and the schedule carried on by design, so the next run re-populates the ledger. No user-visible path and no data loss beyond one ledger entry.

Where to look

  • TriageIssueSweep — the AddTriageIssueSweep schedule; the failing write is TriageIntake.TriageLedger.RecordOrFault against Hosting/Triage/Status, logged at Error in the Catch of the cycle.
  • Hosting/Triage/Status — the ledger node that failed to hydrate; it is healthy now.
  • The inner BaseStateTimeoutException surfaces from MeshWeaver.Mesh.MeshNodeStreamHandle — if this recurs, check the node's per-node hub activation logs and whether RLS is in play for the system-security identity on that path.
Evidence
Fingerprint ff9c336d72f142bd
Category TriageIssueSweep
Severity Error
Exception System.TimeoutException
Namespace memex
Pods memex-portal-deployment-85f748bdd7-nk4wg
Occurrences 1
First seen 2026-09-26 17:42:08Z
Last seen 2026-09-26 17:42:08Z
Routing not determined — no configured route matches the category TriageIssueSweep. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-09-26 17:42:08Z memex-portal-deployment-85f748bdd7-nk4wg fail: TriageIssueSweep[0]
      [TriageSweep] the run (Ok, found 144, created 0, re-opened 0) could NOT be recorded on Hosting/Triage/Status — the Status page does not show it
      System.TimeoutException: Update aborted: no initial state arrived for 'Hosting/Triage/Status' within 30s. Likely causes — (1) RLS silently rejected the prior CreateNode, (2) the path is misspelled / points at a namespace no NodeType claims, (3) the node was deleted between create and update, or (4) the per-node hub activated but its MeshDataSource didn't load the node from persistence. NO change item at all reached the mirror inside the bound — no snapshot ever hydrated. This does NOT establish that the owner never answered: a fresh subscribe is ACKNOWLEDGED BEFORE hydration, so an owner that acked and then produced no frame (a cold data source, a per-node hub still activating, an initial read that yielded nothing) reads identically here. Both the owner's activation/routing AND its initial read are still in scope.
       ---> MeshWeaver.Mesh.MeshNodeStreamHandle+BaseStateTimeoutException: NO change item at all reached the mirror inside the bound — no snapshot ever hydrated. This does NOT establish that the owner never answered: a fresh subscribe is ACKNOWLEDGED BEFORE hydration, so an owner that acked and then produced no frame (a cold data source, a per-node hub still activating, an initial read that yielded nothing) reads identically here. Both the owner's activation/routing AND its initial read are still in scope.
         --- End of inner exception stack trace ---

Opened automatically from Admin/_LogIncident/ff9c336d72f142bd. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site c2d73db02115386d: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
4h 24m
Merged PRs (30d)
961

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.