Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

DataContext hub initialization fails on OIB/LegalEntity nodes: NullReferenceException in TypeDefinition.GetKey during TypeSource/InstanceCollection build

Closed
  • Is something wrong?
#6,645 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
backend

Research direction

Read TypeDefinition.GetKey at src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:128, then trace how InstanceCollection builds its key dictionary and how TypeSource.Initialize supplies instances. The payload does not establish which value is null or provide access to the affected tenant nodes, so first compare the two candidate causes by reading both nodes and retrying activation on memex-cloud. Done means identifying and resolving the cause of the failed hub initialization; this incident is marked as a duplicate of #6643 and may be closed in its favor.

Written by the indexing model from the issue text.

Description

bug sev:M

Observed

Hub initialization failed at the MeshWeaver.Data.DataContext site for two concrete nodes on the memex-cloud instance, pod memex-portal-deployment-5897c49648-jtqxn: OIB/LegalEntity/TEST-OIB-TW-Co and OIB/LegalEntity/TEST-OIB-HK-Ltd. In each case the stack shows MeshWeaver.Data.TypeSource1.Initialize(TypeSource.cs:113) feeding an instance batch to theMeshWeaver.Data.InstanceCollectionconstructor (InstanceCollection.cs:36), which builds anImmutableDictionarykeyed viaMeshWeaver.Messaging.Serialization.TypeDefinition.GetKey(Object instance)(TypeDefinition.cs:128);GetKeythrewNullReferenceException, the initialization aggregated it, and the per-node hub latched **FAILED state**. This log site is the one that carries the node paths unmasked — the sibling incidents for the same event see the path only as {path}`.

The identical stack fired at two sibling log sites at the same instants, both already filed: the MeshWeaver.Graph.CompileWatcher site (05c5ef0f7e92bbd8, GitHub #6643 — five watchers faulting and re-establishing in a loop; its masked path covers both nodes and its loop ran past this site's last occurrence) and the MeshWeaver.Messaging.MessageHub site (e740675ef2d0666e, GitHub #6644 — sync-hub buildup faults). Same event, same pod, same window.

Where and when

One pod, two distinct nodes, two short bursts roughly 6.5 minutes apart: TEST-OIB-TW-Co failing twice within ~9 s at 22:23Z, TEST-OIB-HK-Ltd failing three times within ~15 s at 22:29–22:30Z — activation-retry clusters, not a steady rate. The window coincides with heavy NodeType compile/bake churn and StuckRoll/silo-unavailable alarms on the same deployment that evening; no causal link established. Whether either hub has initialized successfully since is not in the evidence — though #6643's watcher loop was still faulting on TEST-OIB-HK-Ltd as of 22:30:20Z.

Impact

Two OIB LegalEntity nodes — both ids suggest test data — were not served by this pod while their hubs were FAILED. No data loss or corruption is indicated. Bounded blast radius: one pod, one tenant instance, two node paths; the failure was user-visible to anything activating those nodes.

Not established

  • The cause. The stack fixes the dereference site but not what was null: the instance argument handed to GetKey, or a member of the TypeDefinition used to compute the key. The log does not distinguish these.
  • Which TypeSource type parameter the failing initialization served — the DataContext message carries only the node path.
  • Whether the nodes serve today, and whether other pods would fail likewise — only one pod observed.
  • The nodes' content. Neither OIB/LegalEntity/TEST-OIB-TW-Co nor OIB/LegalEntity/TEST-OIB-HK-Ltd is readable from this control instance (Get: not found); the tenant data lives on memex-cloud, so the instances fed to the InstanceCollection could not be inspected from here.

Candidate causes and the reading that separates them

  1. A stored instance under one/both of the nodes' subtrees whose key member — or the object carrying it — is null (data defect). Predicts a deterministic failure on every activation, on every pod, until the data changes. Reading: read the typed instances under the two node paths on memex-cloud for a null key field; a fresh activation failing identically also fits only this cause.
  2. A TypeDefinition for the runtime type not yet (or never) built during a NodeType release adoption — consistent with the [LateReleaseAdoption] watcher faulting in #6643 and the compile/release churn around the burst. Predicts a transient failure: an activation after the release settles succeeds. Reading: activate either node on memex-cloud now; success supports this over (1).
  3. A lazy-initialization race between concurrent TypeSource initializations (five watchers and two sync streams hit the same type concurrently). Predicts rare and non-reproducible in isolation. Reading: line 128 of TypeDefinition.cs in the framework source — which reference is dereferenced there separates instance-null from definition-null outright. (The framework source is not mirrored in this mesh; file/line references are exactly as printed in the logged stacks.)

Readings 1 and 2 are cheap — one node read and one activation on memex-cloud — and between them decide data-defect vs release-window. Both nodes should be checked; the evidence does not establish that the two share one cause.

Duplicate note

This incident is the same event as GitHub #6643 (CompileWatcher site, largest occurrence count, filed) and #6644 (MessageHub site, filed). What this site adds is the concrete node identity for both affected paths, including the second node OIB/LegalEntity/TEST-OIB-HK-Ltd. The ticket can likely be closed in favour of #6643 once its evidence is absorbed.

Where to look

  • MeshWeaver.Messaging.Serialization.TypeDefinition.GetKey — src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:128 (top frame)
  • MeshWeaver.Data.InstanceCollection constructor — src/MeshWeaver.Data.Contract/InstanceCollection.cs:36 (identity selector applied to the batch)
  • MeshWeaver.Data.TypeSource1.Initialize—src/MeshWeaver.Data/TypeSource.cs:113` (batch origin)
  • Nodes OIB/LegalEntity/TEST-OIB-TW-Co and OIB/LegalEntity/TEST-OIB-HK-Ltd on the memex-cloud instance (not reachable from this control instance)
  • Sibling incidents of the same event: 05c5ef0f7e92bbd8 (#6643), e740675ef2d0666e (#6644)
Evidence
Fingerprint 392512339f030b64
Category MeshWeaver.Data.DataContext
Severity Error
Exception System.AggregateException
Top frame MeshWeaver.Messaging.Serialization.TypeDefinition.GetKey(Object instance)
Namespace memex-cloud
Pods memex-portal-deployment-5897c49648-jtqxn
Occurrences 5
First seen 2026-10-12 22:23:19Z
Last seen 2026-10-12 22:30:08Z
Routing not determined — no configured route matches the category MeshWeaver.Data.DataContext. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-12 22:23:19Z memex-portal-deployment-5897c49648-jtqxn fail: MeshWeaver.Data.DataContext[0]
      DataContext initialization failed for OIB/LegalEntity/TEST-OIB-TW-Co. Hub is now in FAILED state.
      System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
       ---> System.NullReferenceException: Object reference not set to an instance of an object.
         at MeshWeaver.Messaging.Serialization.TypeDefinition.GetKey(Object instance) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:line 128
         at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass11_0`3.<ToImmutableDictionary>b__0(TSource element)
         at System.Linq.Enumerable.ArraySelectIterator`2.MoveNext()
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
         at MeshWeaver.Data.InstanceCollection..ctor(IEnumerable`1 instances, Func`2 identity) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data.Contract/InstanceCollection.cs:line 36
         at MeshWeaver.Data.TypeSource`1.<Initialize>b__20_0(IEnumerable`1 items) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/TypeSource.cs:line 113
         at System.Reactive.Linq.ObservableImpl.Select`2.Selector._.OnNext(TSource value)
         --- End of inner exception stack trace ---
2026-10-12 22:23:28Z memex-portal-deployment-5897c49648-jtqxn fail: MeshWeaver.Data.DataContext[0]
      DataContext initialization failed for OIB/LegalEntity/TEST-OIB-TW-Co. Hub is now in FAILED state.
      System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
       ---> System.NullReferenceException: Object reference not set to an instance of an object.
         at MeshWeaver.Messaging.Serialization.TypeDefinition.GetKey(Object instance) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:line 128
         at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass11_0`3.<ToImmutableDictionary>b__0(TSource element)
         at System.Linq.Enumerable.ArraySelectIterator`2.MoveNext()
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
         at MeshWeaver.Data.InstanceCollection..ctor(IEnumerable`1 instances, Func`2 identity) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data.Contract/InstanceCollection.cs:line 36
         at MeshWeaver.Data.TypeSource`1.<Initialize>b__20_0(IEnumerable`1 items) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/TypeSource.cs:line 113
         at System.Reactive.Linq.ObservableImpl.Select`2.Selector._.OnNext(TSource value)
         --- End of inner exception stack trace ---
2026-10-12 22:29:53Z memex-portal-deployment-5897c49648-jtqxn fail: MeshWeaver.Data.DataContext[0]
      DataContext initialization failed for OIB/LegalEntity/TEST-OIB-HK-Ltd. Hub is now in FAILED state.
      System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
       ---> System.NullReferenceException: Object reference not set to an instance of an object.
         at MeshWeaver.Messaging.Serialization.TypeDefinition.GetKey(Object instance) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:line 128
         at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass11_0`3.<ToImmutableDictionary>b__0(TSource element)
         at System.Linq.Enumerable.ArraySelectIterator`2.MoveNext()
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
         at MeshWeaver.Data.InstanceCollection..ctor(IEnumerable`1 instances, Func`2 identity) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data.Contract/InstanceCollection.cs:line 36
         at MeshWeaver.Data.TypeSource`1.<Initialize>b__20_0(IEnumerable`1 items) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/TypeSource.cs:line 113
         at System.Reactive.Linq.ObservableImpl.Select`2.Selector._.OnNext(TSource value)
         --- End of inner exception stack trace ---
2026-10-12 22:29:57Z memex-portal-deployment-5897c49648-jtqxn fail: MeshWeaver.Data.DataContext[0]
      DataContext initialization failed for OIB/LegalEntity/TEST-OIB-HK-Ltd. Hub is now in FAILED state.
      System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
       ---> System.NullReferenceException: Object reference not set to an instance of an object.
         at MeshWeaver.Messaging.Serialization.TypeDefinition.GetKey(Object instance) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:line 128
         at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass11_0`3.<ToImmutableDictionary>b__0(TSource element)
         at System.Linq.Enumerable.ArraySelectIterator`2.MoveNext()
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
         at MeshWeaver.Data.InstanceCollection..ctor(IEnumerable`1 instances, Func`2 identity) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data.Contract/InstanceCollection.cs:line 36
         at MeshWeaver.Data.TypeSource`1.<Initialize>b__20_0(IEnumerable`1 items) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/TypeSource.cs:line 113
         at System.Reactive.Linq.ObservableImpl.Select`2.Selector._.OnNext(TSource value)
         --- End of inner exception stack trace ---
2026-10-12 22:30:08Z memex-portal-deployment-5897c49648-jtqxn fail: MeshWeaver.Data.DataContext[0]
      DataContext initialization failed for OIB/LegalEntity/TEST-OIB-HK-Ltd. Hub is now in FAILED state.
      System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
       ---> System.NullReferenceException: Object reference not set to an instance of an object.
         at MeshWeaver.Messaging.Serialization.TypeDefinition.GetKey(Object instance) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/TypeDefinition.cs:line 128
         at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass11_0`3.<ToImmutableDictionary>b__0(TSource element)
         at System.Linq.Enumerable.ArraySelectIterator`2.MoveNext()
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
         at MeshWeaver.Data.InstanceCollection..ctor(IEnumerable`1 instances, Func`2 identity) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data.Contract/InstanceCollection.cs:line 36
         at MeshWeaver.Data.TypeSource`1.<Initialize>b__20_0(IEnumerable`1 items) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/TypeSource.cs:line 113
         at System.Reactive.Linq.ObservableImpl.Select`2.Selector._.OnNext(TSource value)
         --- End of inner exception stack trace ---

Opened automatically from Admin/_LogIncident/392512339f030b64. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site bebfb32bca80328b: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
5h 34m
Merged PRs (30d)
960

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.