Hub initialization marked FAILED when a shutdown lands mid-init at the StampedFields.LoadDeclaration buildup action, despite the documented shutdown-recognition fix

Read the MessageHub initialization failure handler and the shutdown recognition logic added by the 2026-08-10 fix, then compare them with MeshWeaver.Graph.Security.StampedFields.LoadDeclaration and the cancellation token path. Check sibling issue #5592 and the older probe-dispose incident fingerprints to distinguish a coverage gap from a regression. Done means shutdown during initialization is recognized as normal, without a fail-level log or FAILED state, with a test covering disposal during LoadDeclaration.

Observed

While the node hub for Admin/Purchases/plan-AgenticBusiness-lehmannmartin77 (memex-cloud) was initializing, its buildup action 2 of 3 — StampedFields.LoadDeclaration, the per-activation read of the node type's stamped-fields declaration (Stamped Fields, "What it costs / per activation") — was cancelled with System.OperationCanceledException: the hub was told to shut down before its initialization completed. The MeshWeaver.Messaging.MessageHub failure path then put the hub in FAILED state and logged at fail level.

This is precisely the outcome the platform documented as fixed on 2026-08-10 (Hub shutdown during initialization no longer reported as failure): initialization that ends because the hub (or its parent) is shutting down is meant to be recognized as a normal shutdown — no error log, no FAILED-state residue. The fail-level line and the FAILED state reappeared here.

Where and when

A single occurrence on one portal pod. The incident falls within the hub's own init window being interrupted by a teardown — the exception text is the recognized-shutdown message, not a genuine data or configuration fault. Nothing at the pod level points to a process shutdown or roll at that instant in the Ops log; a per-hub dispose (a recycle, a DisposeRequest, or a parent-space teardown) is the likely trigger, but the source of the shutdown instruction is not on the incident.

Impact

The hub was being torn down regardless of the fault, and the next access to its address would activate a fresh hub, so no user-visible path is blocked by the failure itself. The impact is the documented guarantee being broken: fail-level noise and FAILED-state residue for a routine shutdown race, i.e. exactly the residue the 2026-08-10 fix removed.

Not established

• Why the shutdown was requested — an operator recycle, a DisposeRequest, a parent-space teardown, or a hub-idle disposal. Not on the incident.
• Whether the hub's FAILED state left live residue after the log line (the hub was being disposed; whether the disposal completed cleanly is not recorded here).
• Where exactly the OperationCanceledException was thrown and caught — the incident carries no stack frames, so the path between the cancellation and the fail log in the MessageHub initialization failure handler was not read.
• Whether the same shape has occurred on other pods or at other hub paths under this fingerprint's masking; only one sample is recorded.

Candidate causes and the reading that separates them

1. The shutdown recognition from the 2026-08-10 fix does not cover this buildup action or this exception type. The fix targeted probe hubs racing teardown; StampedFields.LoadDeclaration is a newer buildup action (the Stamped Fields architecture was last rewritten the day before this occurrence), and an OperationCanceledException from the shutdown token may not be classified the same way as the HubDisposingException the fix handled. Predicts: the failure appears only on hubs whose init includes LoadDeclaration and only when a shutdown lands between buildup actions 1 and 3. Discriminating reading: the version history / diff of the recognition logic in the MessageHub initialization failure path versus the introduction of LoadDeclaration; a test that posts a DisposeRequest during LoadDeclaration on a fresh node hub would show whether the OCE reaches the fail log.
2. Same classification gap as the filed sibling #5592, one more surface. Systemorph/MeshWeaver#5592 recorded DataExtensions.StartDataSourcesAndOpenGate faulting with HubDisposingException during a parent teardown race and being marked FAILED; this is the same gap (a shutdown-during-init exception treated as a hard initialization failure) at StampedFields.LoadDeclaration with OperationCanceledException. Predicts: other buildup actions also surface this under a teardown race, each with its own exception type. Discriminating reading: whether #5592's resolution, once landed, also covers the OperationCanceledException path — if it was resolved narrowly for HubDisposingException, this ticket names the second surface; if broadly, this may be a duplicate to close against it.
3. A genuine regression of the 2026-08-10 fix. Predicts: the older probe-hub races (which the fix demonstrably quieted) would also reappear in the error logs around the same deployment. Discriminating reading: the Admin log-incidents at the older ObjectDisposedException/probe-dispose fingerprints (Admin/_LogIncident/4e86cd5956ddddaf) since the current deployment's date — recurrence there would point at a regression, silence there would point at a gap specific to the newer path.

Where to look

• The MessageHub initialization failure handler that logs "Hub {path} initialization failed … Hub is now in FAILED state" and the shutdown-recognition logic installed with the 2026-08-10 fix.
• MeshWeaver.Graph.Security.StampedFields — the LoadDeclaration buildup action and StampedFieldDeclarationHolder (Stamped Fields, "Where it lives").
• The cancellation token path between a hub shutdown instruction and buildup actions in flight.
• Sibling evidence: incident Admin/_LogIncident/6f9895a929bc8b1b (filed as #5592), same category and same FAILED-state outcome at a different buildup action.

The evidence is consistent with all three candidates; the reading that separates 1/2 from 3 is whether the older fingerprints have recurred since this deployment.

Evidence

2026-10-13 00:08:12Z memex-portal-deployment-54856649cb-m2jn9 fail: MeshWeaver.Messaging.MessageHub[0]
      Hub Admin/Purchases/plan-AgenticBusiness-lehmannmartin77 initialization failed — BuildupAction 2 of 3 (StampedFields.LoadDeclaration) faulted (OperationCanceledException: Hub Admin/Purchases/plan-AgenticBusiness-lehmannmartin77 was told to shut down before its initialization completed). Hub is now in FAILED state.
      System.OperationCanceledException: Hub Admin/Purchases/plan-AgenticBusiness-lehmannmartin77 was told to shut down before its initialization completed

_{Opened automatically from Admin/_LogIncident/f7ce6330a2525f2a. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site c4f0569689300af8: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}