Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Message hub discards deferred SubscribeRequests on Orleans deactivation instead of draining them

Open
  • Is something wrong?
#5,718 437 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@rbuergi is already working on this.

Since Oct 11, 2026.

  • #6535 by @rbuergi — open

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
28/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
backend

Research direction

Start in MeshWeaver.Messaging.MessageService, at the Dispose path that handles deferred messages when RunLevel is ShutDown, and inspect log event 7301. Compare that behavior with Orleans deactivation request handling for the hub grain. The issue leaves the intended behavior unresolved: determine whether graceful deactivation should drain deferred work or keep discarding it with a lower-severity, retry-guidance log.

Written by the indexing model from the issue text.

Description

bug sev:L

What is failing

During a cluster-member shutdown, log hub grains (all under Ops/Logs/...) are deactivated while still holding SubscribeRequests deferred behind their DataContextInit and MeshNodeInit gates. MeshWeaver.Messaging.MessageService discards those messages on dispose instead of draining them, so a client's subscribe never takes effect; the sender is answered ShuttingDown.

Probable cause

High confidence in the mechanism, medium in the defect judgement. Orleans deactivates the grain activations because the member server responsible for their directory registration entered ShuttingDown, and the hub's teardown path (RunLevel=ShutDown) drops work still parked behind closed init gates rather than awaiting (or explicitly failing) the gates first. The logging is deliberate and well-attributed, so this is likely by-design drain behaviour — the open question is whether deferred-but-accepted work should be awaited during graceful deactivation, and whether an intentional, sender-acknowledged discard should log at fail severity.

Impact

One 2-second burst (33 occurrences, both portal pods) at 06:58 UTC on 2026-09-24, coinciding with a single server shutdown — not a recurring or ongoing fault. Affected traffic is one cache client (cache/EfNTPfFIy0WLbHUEEmk3jg) whose subscribes to log hubs were rejected; since it receives ShuttingDown it can retry. No portal outage, no data corruption; worst case is a brief window where log subscriptions are not established until the sender retries.

Where to look

MeshWeaver.Messaging.MessageService, log event 7301, the Dispose path that iterates deferred messages when the hub's RunLevel is ShutDown. Compare against the Orleans deactivation request handling for the hub grain — the fix candidate is either draining deferred work behind closed gates during graceful deactivation, or demoting this to warn/info with retry guidance if the discard-and-answer is intended.

Evidence
Fingerprint 4ea4053fbe1ba4fe
Category MeshWeaver.Messaging.MessageService
Severity Error
Namespace memex-cloud
Pods memex-portal-deployment-76cf776847-2x9rq, memex-portal-deployment-76cf776847-9p7z4
Occurrences 33
First seen 2026-09-24 06:58:30Z
Last seen 2026-09-24 06:58:32Z
Routing not determined — no configured route matches the category MeshWeaver.Messaging.MessageService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-09-24 06:58:32Z memex-portal-deployment-76cf776847-9p7z4 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub Ops/Logs/memex-cloud-1790189483100369660-97dc9b-r4nkg is disposing with SubscribeRequest (id=0zbXQRgiZkeB2VHttJbMKg, from cache/EfNTPfFIy0WLbHUEEmk3jg) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by Orleans deactivating grain Ops/Logs/memex-cloud-1790189483100369660-97dc9b-r4nkg; why: DirectoryFailure — This activation is being deactivated because server S10.244.9.20:11111:149237842 entered status ShuttingDown and was responsible for this activation's grain directory registration.. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-24 06:58:32Z memex-portal-deployment-76cf776847-9p7z4 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub Ops/Logs/memex-1790232346594424121-dd76bb-cbbj8 is disposing with SubscribeRequest (id=q0gMjYkc6kCBd_DRx8mKZw, from cache/EfNTPfFIy0WLbHUEEmk3jg) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by Orleans deactivating grain Ops/Logs/memex-1790232346594424121-dd76bb-cbbj8; why: DirectoryFailure — This activation is being deactivated because server S10.244.9.20:11111:149237842 entered status ShuttingDown and was responsible for this activation's grain directory registration.. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-24 06:58:32Z memex-portal-deployment-76cf776847-9p7z4 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub Ops/Logs/memex-cloud-1790187102889628427-4d84f9-mvzb5 is disposing with SubscribeRequest (id=P-r1DXPsoEKvzQXaWPPN0w, from cache/EfNTPfFIy0WLbHUEEmk3jg) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by Orleans deactivating grain Ops/Logs/memex-cloud-1790187102889628427-4d84f9-mvzb5; why: DirectoryFailure — This activation is being deactivated because server S10.244.9.20:11111:149237842 entered status ShuttingDown and was responsible for this activation's grain directory registration.. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-24 06:58:32Z memex-portal-deployment-76cf776847-9p7z4 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub Ops/Logs/memex-cloud-1790187042409595745-4d84f9-n7g6b is disposing with SubscribeRequest (id=TyM2rELHdECGiwyNAmDmDw, from cache/EfNTPfFIy0WLbHUEEmk3jg) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by Orleans deactivating grain Ops/Logs/memex-cloud-1790187042409595745-4d84f9-n7g6b; why: DirectoryFailure — This activation is being deactivated because server S10.244.9.20:11111:149237842 entered status ShuttingDown and was responsible for this activation's grain directory registration.. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.
2026-09-24 06:58:32Z memex-portal-deployment-76cf776847-9p7z4 fail: MeshWeaver.Messaging.MessageService[7301]
      [DISPOSE-DISCARD] Hub Ops/Logs/memex-cloud-1790187042471054791-4d84f9-n7g6b is disposing with SubscribeRequest (id=ZOjs4RhjQ0KJP09iVqaOWg, from cache/EfNTPfFIy0WLbHUEEmk3jg) still deferred; initialization gates closed at deferral: [DataContextInit,MeshNodeInit] — the message is NOT processed; the sender is answered ShuttingDown. RunLevel=ShutDown. This teardown was requested by Orleans deactivating grain Ops/Logs/memex-cloud-1790187042471054791-4d84f9-n7g6b; why: DirectoryFailure — This activation is being deactivated because server S10.244.9.20:11111:149237842 entered status ShuttingDown and was responsible for this activation's grain directory registration.. Accepted work must be drained before a hub goes down — that attribution is who to ask why this hub went down with work still parked behind its gates.

Re-addressed by the current identity function: this incident inherited d2249f800ffc2577 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Opened automatically from Admin/_LogIncident/4ea4053fbe1ba4fe. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 68a7699745069b20: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
4h 19m
Merged PRs (30d)
932

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.