NodeType pre-warm: warm-up update on Admin/Build received no initial state within 30s (bake recorded unproven on one memex-cloud pod)

Read MeshWeaver.Hosting.DynamicTypePreWarmerHostedService and MeshWeaver.Mesh.MeshNodeStreamHandle, starting with the warm-up Update/subscribe path and BaseStateTimeoutException. Compare the pod's Ops/Logs around 16:00:31Z with the Admin/Build version history and tenant-visible persistence row to distinguish the listed causes. Reproduce the timeout or identify its cause, then verify the next warm-up records the bake as proven.

Observed

The DynamicTypePreWarmerHostedService background warm-up stream faulted on one portal pod: its Update against the Admin/Build node (Build/BuildState, the build-coordination node the pre-warmer warms NodeType hubs against) aborted with System.TimeoutException — "no initial state arrived for 'Admin/Build' within 30s" — with inner exception MeshWeaver.Mesh.MeshNodeStreamHandle+BaseStateTimeoutException: no change item at all reached the mirror inside the bound; no snapshot ever hydrated. The pod's warm-up recorded its NodeType bake as unproven rather than complete.

The component degrades by design here: the logged message states that nothing gates on this state, and lazy compilation still works, so correctness of the pod is not in question — only the pre-verified-bake guarantee for that boot.

Where and when

A single occurrence on a single pod (memex-portal-deployment-854d97b779-jsldq) in namespace memex-cloud, at 2026-10-28T16:00:31Z. The Admin/Build node itself is live and healthy: it exists, is Active, and was last modified at 15:47:31Z — about 13 minutes before the timeout — when a fresh bake for framework version c003e001 (223/223 usable across 55 chunks) was recorded ready. So writes and updates on Admin/Build were succeeding shortly before and the node was not deleted.

Impact

One pod booted with an unproven bake in this window; the portal's own warm-up history (Ops/Logs) shows warm-up otherwise completing normally across pods. No user-facing path is evidenced as broken. Judged low severity from the numbers: one occurrence, one pod, a background verification path with designed degradation.

Not established

• The cause. The log site itself enumerates candidates (below) and explicitly notes that a fresh subscribe is acknowledged before hydration, so an owner that acked but produced no frame (cold data source, per-node hub still activating, initial read yielding nothing) reads identically to an owner that never answered — the evidence does not distinguish them.
• Whether the condition persisted or self-cleared: only one occurrence is recorded; nothing shows the outcome of a subsequent warm-up on this pod.
• Whether the same shape occurred on other pods in the same boot window (no other pod appears in this fingerprint).

Candidate causes and the reading that separates them

1. RLS silently rejected the prior CreateNode. Predicts: the Admin/Build row is absent when read under this pod's tenant context, and the failure is systematic for that tenant. Discriminating reading: the persistence row for Admin/Build as seen from that pod's tenant, and whether the failure recurs on the pod's next restart.
2. Path/namespace claim mismatch. Predicts: every warm-up on this image fails identically. Against it: the Admin namespace's Build NodeType demonstrably works — the node has a long version history and was updated at 15:47:31Z. Discriminating reading: whether the pod's next boot reproduces the timeout.
3. Node deleted between create and update. Predicts a gap in Admin/Build's version history around 16:00:31Z. Against it: the node exists now and was last modified at 15:47:31Z. Discriminating reading: Admin/Build's version history at that instant.
4. Per-node hub activated but its MeshDataSource produced no initial frame (cold data source, hub still activating, or an initial read that yielded nothing — all read identically here because the subscribe ack precedes hydration). Predicts: a transient — a restart or a later access re-hydrates fine, and the warm-up logs of this pod show a later successful run. Discriminating reading: the owner-side hub logs for Admin/Build on this pod at 16:00:31Z, and the outcome of the pod's next warm-up sweep.

The most similar prior incident at this component is Admin/_LogIncident/3cd719ae33cd1e19 (filed as issue #1635, August 2026), whose samples include this exact "no initial state arrived for 'Admin/Build' within 30s" shape — but that incident folded several different timeout shapes (hub-request timeouts, generic operation timeouts) under one fingerprint, so this is related history rather than a duplicate. Admin/_LogIncident/aa02f5158f773ec0 is a different failure mode at the same node (MeshNode Unknown on an Npgsql failure).

Where to look

• MeshWeaver.Hosting.DynamicTypePreWarmerHostedService (namespace MeshWeaver.Hosting, repository Systemorph/MeshWeaver) — the warm-up sweep and its Update/subscribe against Admin/Build.
• MeshWeaver.Mesh.MeshNodeStreamHandle — BaseStateTimeoutException and the subscribe-ack-before-hydration path.
• The Admin/Build node (Admin/Build, nodeType Build) — version history around 2026-10-28T16:00Z.
• This pod's warm-up logs in Ops/Logs for pod memex-portal-deployment-854d97b779-jsldq before and after 16:00:31Z.

Evidence

2026-10-28 16:00:31Z memex-portal-deployment-854d97b779-jsldq fail: MeshWeaver.Hosting.DynamicTypePreWarmerHostedService[0]
      DynamicTypePreWarmer: warm-up stream faulted — this pod verified NOTHING about its NodeTypes (lazy compile still works, so it stays correct). Nothing gates on this state here; the bake is recorded as unproven rather than complete.
      System.TimeoutException: Update aborted: no initial state arrived for 'Admin/Build' within 30s. Likely causes — (1) RLS silently rejected the prior CreateNode, (2) the path is misspelled / points at a namespace no NodeType claims, (3) the node was deleted between create and update, or (4) the per-node hub activated but its MeshDataSource didn't load the node from persistence. NO change item at all reached the mirror inside the bound — no snapshot ever hydrated. This does NOT establish that the owner never answered: a fresh subscribe is ACKNOWLEDGED BEFORE hydration, so an owner that acked and then produced no frame (a cold data source, a per-node hub still activating, an initial read that yielded nothing) reads identically here. Both the owner's activation/routing AND its initial read are still in scope.
       ---> MeshWeaver.Mesh.MeshNodeStreamHandle+BaseStateTimeoutException: NO change item at all reached the mirror inside the bound — no snapshot ever hydrated. This does NOT establish that the owner never answered: a fresh subscribe is ACKNOWLEDGED BEFORE hydration, so an owner that acked and then produced no frame (a cold data source, a per-node hub still activating, an initial read that yielded nothing) reads identically here. Both the owner's activation/routing AND its initial read are still in scope.
         --- End of inner exception stack trace ---

_{Opened automatically from Admin/_LogIncident/800cb28cfa54d3f9. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 5aae9031f835d109: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}