Two sync-hub buildups fault with NullReferenceException in TypeDefinition.GetKey during TypeSource initialization for OIB/LegalEntity/TEST-OIB-TW-Co

Start by identifying which subscribers the two sync hub addresses serve, then read TypeDefinition.GetKey at src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:128, InstanceCollection at src/MeshWeaver.Data.Contract/InstanceCollection.cs:36, and TypeSource.Initialize at src/MeshWeaver.Data/TypeSource.cs:113. Check the node OIB/LegalEntity/TEST-OIB-TW-Co on memex-cloud and activate it again to distinguish a data defect from a transient initialization failure. Done means establishing the cause and whether the node and sync streams recover; the sibling incident already has a filed draft covering this event.

Observed

Two distinct per-stream synchronization sub-hubs — sync/IWdeAmdPzEWWsYMJI9DFqw and sync/hVUtzzdp50eQZtLc522qUg — each failed initialization during the SynchronizationStream1.b__105_11buildup action (1 of 1) with aSystem.NullReferenceExceptionthrown fromMeshWeaver.Messaging.Serialization.TypeDefinition.GetKey(Object instance) (src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:128), reached via the MeshWeaver.Data.InstanceCollection constructor (src/MeshWeaver.Data.Contract/InstanceCollection.cs:36) building its ImmutableDictionary, from MeshWeaver.Data.TypeSource1.<Initialize>b__20_0 (src/MeshWeaver.Data/TypeSource.cs:113). Both hubs latched FAILED.

These two failures are not isolated to this log site. The same instants produced sibling incidents with the identical stack: the MeshWeaver.Data.DataContext site (392512339f030b64) names the concrete node — OIB/LegalEntity/TEST-OIB-TW-Co — and the MeshWeaver.Graph.CompileWatcher site (05c5ef0f7e92bbd8) records five watchers (compile, ReleaseRequest, [LateReleaseAdoption], sources, adopted source-stamp) faulting and re-establishing in a loop on that same node path. The two sync-hub failures here are that same event surfacing at the sync-stream buildup: a TypeSource initializing against the data of TEST-OIB-TW-Co could not compute an instance's key, and every activation touching it — two subscriber streams, the node's DataContext, and its compile/release watchers — faulted identically. Cause not established beyond the stack.

Where and when

One pod of the memex-cloud portal deployment, two stream activations nine seconds apart — a burst, not a steady rate. The window coincides with heavy NodeType compile/bake churn and silo-unavailable alarms on the same deployment earlier that evening (open incidents for a NodeType PARKED after compile failure and nodetype_bake health checks Unhealthy); no causal link is established. Whether the node recovered once the watcher loop ended is not in the evidence.

Impact

The node OIB/LegalEntity/TEST-OIB-TW-Co (by its id, test data) was not served while its hub(s) were FAILED, and the two sync streams subscribing to it failed rather than receiving data. One pod, one node, no other namespace — bounded blast radius; no indication of data loss or corruption.

Not established

• The cause. The stack fixes where the null dereference happened (GetKey), not what was null: the instance argument (a null/degraded element in the source enumeration handed to the InstanceCollection ctor) or a member of the TypeDefinition used to compute the key. The log does not distinguish these.
• Which TypeSource type parameter the failing initialization served; the sync hub addresses do not name it.
• Whether it still reproduces on activation, and whether the node serves today.
• Whether other pods would show it — only one pod observed.

Candidate causes and the reading that separates them

1. A stored instance under the node whose key member (or the object carrying it) is null — a data defect. Predicts a deterministic failure on every activation, on every replica, until the data is corrected. Reading: read the content / typed instances under OIB/LegalEntity/TEST-OIB-TW-Co on the memex-cloud instance for a null key field; a fresh activation failing identically also fits only this cause.
2. A TypeDefinition for the runtime type not yet (or never) built — a mid-release-adoption type-shape mismatch, consistent with the [LateReleaseAdoption] watcher faulting in the sibling incident and the compile/release churn around it. Predicts a transient failure: an activation after the release settles succeeds. Reading: activate the same node now; success supports this over (1).
3. A lazy-initialization race between concurrent TypeSource initializations. Predicts rare and non-reproducible in isolation. Reading: line 128 of TypeDefinition.cs — which reference is dereferenced there settles instance-null vs definition-null outright (the framework source is not mirrored in this mesh; the file/line are as printed in the logged stack).

Readings 1 and 2 are cheap (one node read / one activation on memex-cloud) and between them decide data-defect vs release-window.

Duplicate note

Sibling incident 05c5ef0f7e92bbd8 (CompileWatcher site) records the same event with the largest occurrence count and already carries a filed draft covering it, including both siblings; this ticket is the same event at the MessageHub sync-hub site and can likely be closed in its favour. 392512339f030b64 (DataContext site) is the same event as well; its triage errored without filing a draft.

Where to look

• MeshWeaver.Messaging.Serialization.TypeDefinition.GetKey — src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:128 (top frame, per the stack)
• MeshWeaver.Data.InstanceCollection constructor — src/MeshWeaver.Data.Contract/InstanceCollection.cs:36 (where the identity selector is applied to the batch)
• MeshWeaver.Data.TypeSource1.Initialize—src/MeshWeaver.Data/TypeSource.cs:113` (where the instance batch originates)
• The node OIB/LegalEntity/TEST-OIB-TW-Co on the memex-cloud instance (not reachable from this control instance)
• The two hub addresses sync/IWdeAmdPzEWWsYMJI9DFqw and sync/hVUtzzdp50eQZtLc522qUg — identifying which streams/subscribers they serve is the first read that turns this from a signature into a case.

Severity judged sev:M: a clear defect with a bounded blast radius (one test-data node, one pod, one burst); whether the node serves today is the fact that would let a human re-rule it.

Evidence

2026-10-12 22:23:19Z memex-portal-deployment-5897c49648-jtqxn fail: MeshWeaver.Messaging.MessageHub[0]
      Hub sync/IWdeAmdPzEWWsYMJI9DFqw initialization failed — BuildupAction 1 of 1 (SynchronizationStream`1.<ConfigureSynchronizationHub>b__105_11) faulted (NullReferenceException: Object reference not set to an instance of an object.). Hub is now in FAILED state.
      System.NullReferenceException: Object reference not set to an instance of an object.
         at MeshWeaver.Messaging.Serialization.TypeDefinition.GetKey(Object instance) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:line 128
         at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass11_0`3.<ToImmutableDictionary>b__0(TSource element)
         at System.Linq.Enumerable.ArraySelectIterator`2.MoveNext()
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
         at MeshWeaver.Data.InstanceCollection..ctor(IEnumerable`1 instances, Func`2 identity) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data.Contract/InstanceCollection.cs:line 36
         at MeshWeaver.Data.TypeSource`1.<Initialize>b__20_0(IEnumerable`1 items) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/TypeSource.cs:line 113
         at System.Reactive.Linq.ObservableImpl.Select`2.Selector._.OnNext(TSource value)
2026-10-12 22:23:28Z memex-portal-deployment-5897c49648-jtqxn fail: MeshWeaver.Messaging.MessageHub[0]
      Hub sync/hVUtzzdp50eQZtLc522qUg initialization failed — BuildupAction 1 of 1 (SynchronizationStream`1.<ConfigureSynchronizationHub>b__105_11) faulted (NullReferenceException: Object reference not set to an instance of an object.). Hub is now in FAILED state.
      System.NullReferenceException: Object reference not set to an instance of an object.
         at MeshWeaver.Messaging.Serialization.TypeDefinition.GetKey(Object instance) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:line 128
         at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass11_0`3.<ToImmutableDictionary>b__0(TSource element)
         at System.Linq.Enumerable.ArraySelectIterator`2.MoveNext()
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
         at MeshWeaver.Data.InstanceCollection..ctor(IEnumerable`1 instances, Func`2 identity) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data.Contract/InstanceCollection.cs:line 36
         at MeshWeaver.Data.TypeSource`1.<Initialize>b__20_0(IEnumerable`1 items) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/TypeSource.cs:line 113
         at System.Reactive.Linq.ObservableImpl.Select`2.Selector._.OnNext(TSource value)

_{Opened automatically from Admin/_LogIncident/e740675ef2d0666e. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 0d18e4505ad785d1: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}