Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Message routing silently drops deliveries when payload serialization OOMs in ObjectPolymorphicConverter.Write

Chiusa
  • Qualcosa non va?
#6,152 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
28/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp, json
Ambito
backend, distributed-systems

Direzione di ricerca

Read src/MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs around Write (lines 435/448), then trace how OrleansRoutingService and IRoutingGrain.RouteMessage handle serialization failures. Compare payload sizing, memory pressure, and the related incident Admin/_LogIncident/afc6241b330977b0 / MeshWeaver#2194. Done means oversized or failed serialization no longer silently loses deliveries, with behavior covered by relevant tests; the issue does not identify those tests.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:H

What is failing

The Orleans message-routing service (MeshWeaver.Connection.*.OrleansRoutingService) drops message delivery when serialization of the polymorphic message body throws System.OutOfMemoryException. The OOM occurs inside ObjectPolymorphicConverter.Write (MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs, the JsonSerializer.WriteBytesAsObject path), and the routing service logs the failure and moves on — the delivery is lost.

Probable cause (medium-high confidence)

The stack runs Utf8JsonWriter.WriteStringEscapeValue → Utf8JsonWriter.Grow → SharedArrayPool.Rent → GC.AllocateNewArray: the JSON writer ran out of memory while growing its buffer to emit a large string property of the payload. Two readings, which most likely combine:

  1. Oversized payload (likely trigger). One failed delivery targets a CI build-log document (Admin/content/_Documents/logs-ci-Systemorph-MeshWeaver-…log) — routing such content embeds a very large string in the message, and the serialization path has no size cap. The other targets (a portal reads stream, Hosting/Deployment) are ordinary, which fits one oversized message poisoning whatever is in flight.
  2. Pod memory pressure (likely amplifier). Both affected pods also appear in the concurrent GC-bound health-check incident (Admin/_LogIncident/afc6241b330977b0, which references known issue MeshWeaver#2194), and the pod that threw the latest sample was reporting GC pause shares of 55–96% in the minutes beforehand. Under that pressure, even a moderately large payload fails to allocate.

Cause not certain beyond this: capturing the payload being routed at the next burst will settle which reading dominates. This is probably a downstream symptom of the broader memory problem rather than a standalone defect — but the unbounded serialization and the silent drop are both worth fixing regardless.

Impact

Message delivery to the affected targets is silently dropped, so clients on those streams miss updates. The burst is small and fresh (a handful of occurrences on two memex-portal pods within one day), but the same pods are concurrently reporting GC-bound health checks and routing back-pressure, so the deployment is already memory-stressed; an OOM inside the portal's serialization path is a pod-stability risk if it recurs.

Where to look

  • src/MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs, Write (~lines 435/448) — add a payload-size guard (reject or chunk oversized bodies) or stream the serialization instead of buffering via WriteBytesAsObject.
  • The delivery path in OrleansRoutingService / IRoutingGrain.RouteMessage that calls it — decide whether a serialization failure should drop the message silently or fail the delivery with a retry/backoff.
  • Container memory limits vs. live heap on the memex-portal pods, together with the related GC-bound incident (Admin/_LogIncident/afc6241b330977b0, MeshWeaver#2194) — if pod memory pressure is the amplifier, resolving that may clear this symptom.

Related

  • Not a duplicate of any existing incident for this component, but likely shares a root cause with Admin/_LogIncident/afc6241b330977b0 (GC-bound health checks, references MeshWeaver#2194) and coincides in time with Admin/_LogIncident/2e7fbf50d41731c0 (routing back-pressure on the same deployment).
Evidence
Fingerprint fbd2d04d8cc557b0
Category MeshWeaver.Connection.Orleans.OrleansRoutingService
Severity Error
Exception System.OutOfMemoryException
Top frame MeshWeaver.Messaging.Serialization.ObjectPolymorphicConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options)
Namespace memex
Pods memex-portal-deployment-6d8c9665cc-jltnc, memex-portal-deployment-6cfcf78898-gkkdv
Occurrences 3
First seen 2026-10-05 07:43:31Z
Last seen 2026-10-05 14:50:27Z
Routing not determined — no configured route matches the category MeshWeaver.Connection.Orleans.OrleansRoutingService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-05 07:43:31Z memex-portal-deployment-6d8c9665cc-jltnc fail: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Failed to deliver to portal/reads-tPgiR1cLJEGnorK5nPDWkg
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.<AllocateUninitializedArray>g__AllocateNewArrayWorker|77_0[T](Int32 length, Boolean pinned)
         at System.Buffers.SharedArrayPool`1.Rent(Int32 minimumLength)
         at System.Net.ArrayBuffer.EnsureAvailableSpaceCore(Int32 byteCount)
         at System.Text.Json.Utf8JsonWriter.Grow(Int32 requiredSize)
         at System.Text.Json.Utf8JsonWriter.WriteStringMinimized(ReadOnlySpan`1 escapedValue)
         at System.Text.Json.Utf8JsonWriter.WriteStringEscapeValue(ReadOnlySpan`1 value, Int32 firstEscapeIndexVal)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWriteAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsObject(Utf8JsonWriter writer, Object rootValue)
         at System.Text.Json.JsonSerializer.WriteBytesAsObject(Object value, JsonTypeInfo jsonTypeInfo)
         at MeshWeaver.Messaging.Serialization.ObjectPolymorphicConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs:line 435
      …[truncated]
2026-10-05 07:43:37Z memex-portal-deployment-6d8c9665cc-jltnc fail: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Failed to deliver to Hosting/Deployment
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.<AllocateUninitializedArray>g__AllocateNewArrayWorker|77_0[T](Int32 length, Boolean pinned)
         at System.Text.Json.Utf8JsonWriter.WriteStringEscapeValue(ReadOnlySpan`1 value, Int32 firstEscapeIndexVal)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWriteAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsObject(Utf8JsonWriter writer, Object rootValue)
         at System.Text.Json.JsonSerializer.WriteBytesAsObject(Object value, JsonTypeInfo jsonTypeInfo)
         at MeshWeaver.Messaging.Serialization.ObjectPolymorphicConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs:line 435
         at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at Sys…[truncated]
2026-10-05 14:50:27Z memex-portal-deployment-6cfcf78898-gkkdv fail: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Failed to deliver to Admin/content/_Documents/logs-ci-Systemorph-MeshWeaver-111777921171.log
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.<AllocateUninitializedArray>g__AllocateNewArrayWorker|77_0[T](Int32 length, Boolean pinned)
         at System.Buffers.SharedArrayPool`1.Rent(Int32 minimumLength)
         at System.Text.Json.Utf8JsonWriter.WriteStringEscapeValue(ReadOnlySpan`1 utf8Value, Int32 firstEscapeIndexVal)
         at System.Text.Json.JsonDocument.WriteComplexElement(Int32 index, Utf8JsonWriter writer)
         at System.Text.Json.JsonDocument.WriteElementTo(Int32 index, Utf8JsonWriter writer)
         at MeshWeaver.Messaging.Serialization.ObjectPolymorphicConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs:line 448
         at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at Orleans.Serialization.JsonCodec.Orleans.Serialization.Cloning.IDeepCopier.DeepCopy(Object input, CopyContext context) in /_/src/Orleans.Serialization.SystemTextJson/JsonCodec.cs:line 230
         at OrleansCodeGen.MeshWeaver.Connection.Orleans.Proxy_IRoutingGrain.global::MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(IMessageDelivery arg0) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Connection.Orleans/obj/container/linux-x64/Release/net10.0/Orleans.CodeGenerator/Orleans.CodeGenerator.OrleansSerializationSourceGenerator/MeshWeaver.Connecti…[truncated]

Opened automatically from Admin/_LogIncident/fbd2d04d8cc557b0. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site e763b3f945f48342: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
3h 59m
PR unite (30g)
975

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.