Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Message handler for a sync hub dies with OutOfMemoryException and the delivery-pipeline log hides the allocation site

Closed
  • Is something wrong?
#6,160 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp
Domain
backend

Research direction

Start with src/MeshWeaver.Messaging.Hub/MessageService.cs, especially RunHandler around line 2565, and trace how the delivery is available in the catch-all exception handler. Check whether the sync path enforces the transport size limit and compare the oversized-message diagnostics. Done means the log includes the delivery's message type and payload size, with tests covering the added diagnostics; the underlying OOM cause is uncertain.

Written by the indexing model from the issue text.

Description

bug sev:M

What is failing

The message delivery pipeline (MeshWeaver.Messaging.MessageService.RunHandler, the catch-all that runs a hub's message handlers over the Rx delivery stream) reported an unhandled System.OutOfMemoryException while executing a handler for hub sync/-QF_gp2iLEKTpPIdfoRaIA on the memex portal. The delivery's handler died mid-execution and that message was not processed.

Probable cause

Cause unclear — and the evidence limits what can be said. The OOM was logged at RunHandler (MessageService.cs:line 2565), but that frame is the pipeline's exception boundary: the allocation that failed happened inside the handler invocation, below System.Reactive ObservableImpl.Defer.Run, and no deeper frames were captured. So the component at fault is the handler for whatever message the sync hub was processing, not RunHandler itself. Two candidate causes stand out, in order of likelihood: (1) a genuinely oversized sync payload being materialised by a handler — the platform has a documented history of oversized internal messages (refused at send since late August, per An impossibly large message now fails instead of hanging) and of OOMs at large-buffer allocation sites (issue #1351, OOM in MessageDeliveryConverter serialization; issue #5501, OOM in MapPublish's MemoryStream buffering); or (2) transient pod-level memory pressure that surfaced at the next allocation, whatever it was. Confidence: low-medium; this ticket is as much about the observability gap — the unhandled-exception log should capture the delivery's message type and size, and ideally more of the inner stack — as about a specific defect.

Impact

A single occurrence on one portal pod, seconds before this incident was opened — one message delivery lost. Not an outage; but a delivery-pipeline OOM is a silent message-loss mode, and if it recurs on sync traffic it will look like a space that stopped syncing, with only this generic line to go on.

Where to look

src/MeshWeaver.Messaging.Hub/MessageService.cs, RunHandler (~line 2565). Start from the hub address sync/-QF_gp2iLEKTpPIdfoRaIA — correlate what sync operation was running at that timestamp and what message was in flight. Then: (a) enrich the unhandled-exception log in RunHandler with the delivery's message type and payload size so the next occurrence points at the producer, mirroring the refusal diagnostics introduced for oversized messages; (b) check whether the transport's size limit is enforced on the sync path as it is on the send path. Related prior OOMs in the same family, for pattern context: issue #1351 (MemoryAdapterFactory.QueueMessageBatchAsync serialization OOM, incident Admin/_LogIncident/4a50cc9363466ae2) and issue #5501 (MapPublish buffering OOM, incident Admin/_LogIncident/7bb4ddf387689fad) — same broad symptom class, different log sites, so not duplicates of this fingerprint. Architecture context on the delivery pipeline: Debugging Message Flow, Message-Based Communication.

Evidence
Fingerprint aee0aebf6d9f1763
Category MeshWeaver.Messaging.MessageService
Severity Error
Exception System.OutOfMemoryException
Top frame MeshWeaver.Messaging.MessageService.RunHandler(IMessageDelivery delivery)
Namespace memex
Pods memex-portal-deployment-6cfcf78898-gkkdv
Occurrences 1
First seen 2026-10-05 16:32:38Z
Last seen 2026-10-05 16:32:38Z
Routing not determined — no configured route matches the category MeshWeaver.Messaging.MessageService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-05 16:32:38Z memex-portal-deployment-6cfcf78898-gkkdv fail: MeshWeaver.Messaging.MessageService[0]
      Unhandled exception in delivery pipeline for hub sync/-QF_gp2iLEKTpPIdfoRaIA
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at MeshWeaver.Messaging.MessageService.RunHandler(IMessageDelivery delivery) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/MessageService.cs:line 2565
         at System.Reactive.Linq.ObservableImpl.Defer`1._.Run()

Opened automatically from Admin/_LogIncident/aee0aebf6d9f1763. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site de336e74e276acee: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
4h 1m
Merged PRs (30d)
979

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.