Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

MessageService delivery pipeline still OOMs under process-wide heap exhaustion — third recurrence of #4824/#5555, containment fix still parked

Chiusa
  • Qualcosa non va?
#6,161 11 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
15/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
backend, distributed-systems

Direzione di ricerca

Start with src/MeshWeaver.Messaging.Hub/MessageService.cs, especially DrainLoop() and the per-delivery subscribe path; the issue notes this file is about 227k characters and difficult to edit. Read the #5555 investigation and PR #5931 for the suspected allocation drivers, then compare with the recurrence evidence from #4824 and #5555. Done means identifying and addressing the process-wide allocation leak and ensuring a failed delivery does not terminate the drain; the issue indicates this needs investigation and may be folded into the earlier investigation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:H

What is failing

The MeshWeaver.Messaging.MessageService delivery pipeline (MessageService.DrainLoop()) throws System.OutOfMemoryException while subscribing a delivery for a hub; the affected delivery is lost and not retried. This is the third burst of the same defect in three weeks: Systemorph/MeshWeaver#4824 (2026-09-19, 4 occurrences) and #5555 (2026-09-23, recurrence of #4824) showed the identical category, message, top frame and stack shape. The containment fix attempted under #5555 was parked without a review verdict — no PR ever landed — so the defect is unchanged in production.

Probable cause

Medium-to-high confidence, carried over from the #5555 diagnosis which this burst corroborates: the OOM is process-wide heap exhaustion, and the drain loop is merely the first allocation to fail — a victim, not the leak. Evidence: (a) the stacks run entirely through System.Reactive SelectMany/Producer.SubscribeRaw machinery with no application frame above it, identical to the earlier bursts; (b) in #5555 sibling OOMs hit the same pod within seconds at unrelated sites (e.g. MessageHub.Dispose), which a single bad allocation cannot explain; (c) this burst again spans unrelated hub paths on different pods and days. The open allocation drivers named in PR #5931 (Func<JsonDerivedType,bool> and Autofac IResolvePipelineBuilder handlers — PR #5925 removed only the per-hub TypeDefinition driver) remain the leading suspects for the leak itself. The secondary defect — one delivery's exception ending the drain rather than being contained per-delivery — also remains unfixed, since the containment PR was parked.

Impact

Three occurrences across two pods over ~21.5 hours (2026-10-04 19:10 UTC → 2026-10-05 16:32 UTC) in namespace memex. Intermittent, but each burst means a lost delivery and a pod whose heap is exhausted enough that allocation is failing anywhere — i.e. a degrading pod until restart. Notably, the hubs named in this burst's samples are the automated triage machinery itself (worker threads Hosting/Triage/_Thread/bug-systemorph-meshweaver-plugins-2403 and …-5862, plus a CI log document part under Admin/content/_Documents/logs-ci-…log/_DocumentPart/005158): the failure is now disrupting the very pipeline that handles and files these incidents.

Where to look

  • src/MeshWeaver.Messaging.Hub/MessageService.cs — DrainLoop() and the per-delivery subscribe path; the same starting point as #4824/#5555. Note the file is ~227k characters and the fix lane could not edit it — the containment work needs a person or a partial-class split first (see the round-17 note on the #5555 triage item).
  • The allocation drivers named in PR #5931 — these, not the drain loop, are the likely leak.
  • Full prior investigation and evidence: Triage item for #5555 (recurrence chain #4824 → #5555 → this) and Debugging Message Flow for the fate-trail tooling.
  • Likely duplicate of #4824/#5555 — a human can fold this recurrence into that investigation in seconds, but it also signals the parked fix needs attention.
Evidence
Fingerprint f6a4dfaef500b486
Category MeshWeaver.Messaging.MessageService
Severity Error
Exception System.OutOfMemoryException
Top frame MeshWeaver.Messaging.MessageService.DrainLoop()
Namespace memex
Pods memex-portal-deployment-76dd69c7fc-xzsdl, memex-portal-deployment-6cfcf78898-gkkdv
Occurrences 3
First seen 2026-10-04 19:10:46Z
Last seen 2026-10-05 16:32:38Z
Routing not determined — no configured route matches the category MeshWeaver.Messaging.MessageService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-04 19:10:46Z memex-portal-deployment-76dd69c7fc-xzsdl fail: MeshWeaver.Messaging.MessageService[0]
      Unhandled exception in delivery pipeline for hub Hosting/Triage/_Thread/bug-systemorph-meshweaver-plugins-2403
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)
         at System.ObservableExtensions.SubscribeSafe[T](IObservable`1 source, IObserver`1 observer)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)
         at System.ObservableExtensions.SubscribeSafe[T](IObservable`1 source, IObserver`1 observer)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)
         at System.ObservableExtensions.SubscribeSafe[T](IObservable`1 source, IObserver`1 observer)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)
         at System.ObservableExtensions.SubscribeSafe[T](IObservable`1 source, IObserver`1 observer)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)
         at System.ObservableExtensions.SubscribeSafe[T](IObservable`1 source, IObserver`1 observer)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)
         at System.ObservableExtensions.SubscribeSafe[T](IObservable`1 source, IObserver`1 observer)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)
         at System.ObservableExtensions.SubscribeSafe[T](IObservable`1 source, IObserver`1 observer)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)
         at System.ObservableExtensions.SubscribeSafe[T](IObservable`1 source, IObserver`1 observer)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector._.OnNext(TSource value)…[truncated]
2026-10-04 19:10:52Z memex-portal-deployment-76dd69c7fc-xzsdl fail: MeshWeaver.Messaging.MessageService[0]
      Unhandled exception in delivery pipeline for hub Hosting/Triage/_Thread/bug-systemorph-meshweaver-5862
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 obse…[truncated]
2026-10-05 16:32:38Z memex-portal-deployment-6cfcf78898-gkkdv fail: MeshWeaver.Messaging.MessageService[0]
      Unhandled exception in delivery pipeline for hub Admin/content/_Documents/logs-ci-Systemorph-MeshWeaver-111819915277.log/_DocumentPart/005158
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector.CreateSink(IObserver`1 observer)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector.Run(_ sink)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector.Run(_ sink)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector.Run(_ sink)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector.Run(_ sink)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector.Run(_ sink)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector.Run(_ sink)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector.Run(_ sink)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observer, Boolean enableSafeguard)
         at System.Reactive.Linq.ObservableImpl.SelectMany`2.ObservableSelector.Run(_ sink)
         at System.Reactive.Producer`2.SubscribeRaw(IObserver`1 observe…[truncated]

Opened automatically from Admin/_LogIncident/f6a4dfaef500b486. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site cd84fed0e995cdaa: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
6h 28m
PR unite (30g)
951

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.