Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Message routing silently drops deliveries when payload serialization OOMs in ObjectPolymorphicConverter.Write

Open
  • Is something wrong?
#6,152 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
28/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
csharp, json
Domain
backend, distributed-systems

Research direction

Read src/MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs around Write (lines 435/448), then trace how OrleansRoutingService and IRoutingGrain.RouteMessage handle serialization failures. Compare payload sizing, memory pressure, and the related incident Admin/_LogIncident/afc6241b330977b0 / MeshWeaver#2194. Done means oversized or failed serialization no longer silently loses deliveries, with behavior covered by relevant tests; the issue does not identify those tests.

Written by the indexing model from the issue text.

Description

bug sev:H

What is failing

The Orleans message-routing service (MeshWeaver.Connection.*.OrleansRoutingService) drops message delivery when serialization of the polymorphic message body throws System.OutOfMemoryException. The OOM occurs inside ObjectPolymorphicConverter.Write (MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs, the JsonSerializer.WriteBytesAsObject path), and the routing service logs the failure and moves on — the delivery is lost.

Probable cause (medium-high confidence)

The stack runs Utf8JsonWriter.WriteStringEscapeValue → Utf8JsonWriter.Grow → SharedArrayPool.Rent → GC.AllocateNewArray: the JSON writer ran out of memory while growing its buffer to emit a large string property of the payload. Two readings, which most likely combine:

  1. Oversized payload (likely trigger). One failed delivery targets a CI build-log document (Admin/content/_Documents/logs-ci-Systemorph-MeshWeaver-…log) — routing such content embeds a very large string in the message, and the serialization path has no size cap. The other targets (a portal reads stream, Hosting/Deployment) are ordinary, which fits one oversized message poisoning whatever is in flight.
  2. Pod memory pressure (likely amplifier). Both affected pods also appear in the concurrent GC-bound health-check incident (Admin/_LogIncident/afc6241b330977b0, which references known issue MeshWeaver#2194), and the pod that threw the latest sample was reporting GC pause shares of 55–96% in the minutes beforehand. Under that pressure, even a moderately large payload fails to allocate.

Cause not certain beyond this: capturing the payload being routed at the next burst will settle which reading dominates. This is probably a downstream symptom of the broader memory problem rather than a standalone defect — but the unbounded serialization and the silent drop are both worth fixing regardless.

Impact

Message delivery to the affected targets is silently dropped, so clients on those streams miss updates. The burst is small and fresh (a handful of occurrences on two memex-portal pods within one day), but the same pods are concurrently reporting GC-bound health checks and routing back-pressure, so the deployment is already memory-stressed; an OOM inside the portal's serialization path is a pod-stability risk if it recurs.

Where to look

  • src/MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs, Write (~lines 435/448) — add a payload-size guard (reject or chunk oversized bodies) or stream the serialization instead of buffering via WriteBytesAsObject.
  • The delivery path in OrleansRoutingService / IRoutingGrain.RouteMessage that calls it — decide whether a serialization failure should drop the message silently or fail the delivery with a retry/backoff.
  • Container memory limits vs. live heap on the memex-portal pods, together with the related GC-bound incident (Admin/_LogIncident/afc6241b330977b0, MeshWeaver#2194) — if pod memory pressure is the amplifier, resolving that may clear this symptom.

Related

  • Not a duplicate of any existing incident for this component, but likely shares a root cause with Admin/_LogIncident/afc6241b330977b0 (GC-bound health checks, references MeshWeaver#2194) and coincides in time with Admin/_LogIncident/2e7fbf50d41731c0 (routing back-pressure on the same deployment).
Evidence
Fingerprint fbd2d04d8cc557b0
Category MeshWeaver.Connection.Orleans.OrleansRoutingService
Severity Error
Exception System.OutOfMemoryException
Top frame MeshWeaver.Messaging.Serialization.ObjectPolymorphicConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options)
Namespace memex
Pods memex-portal-deployment-6d8c9665cc-jltnc, memex-portal-deployment-6cfcf78898-gkkdv
Occurrences 3
First seen 2026-10-05 07:43:31Z
Last seen 2026-10-05 14:50:27Z
Routing not determined — no configured route matches the category MeshWeaver.Connection.Orleans.OrleansRoutingService. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-05 07:43:31Z memex-portal-deployment-6d8c9665cc-jltnc fail: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Failed to deliver to portal/reads-tPgiR1cLJEGnorK5nPDWkg
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.<AllocateUninitializedArray>g__AllocateNewArrayWorker|77_0[T](Int32 length, Boolean pinned)
         at System.Buffers.SharedArrayPool`1.Rent(Int32 minimumLength)
         at System.Net.ArrayBuffer.EnsureAvailableSpaceCore(Int32 byteCount)
         at System.Text.Json.Utf8JsonWriter.Grow(Int32 requiredSize)
         at System.Text.Json.Utf8JsonWriter.WriteStringMinimized(ReadOnlySpan`1 escapedValue)
         at System.Text.Json.Utf8JsonWriter.WriteStringEscapeValue(ReadOnlySpan`1 value, Int32 firstEscapeIndexVal)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWriteAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsObject(Utf8JsonWriter writer, Object rootValue)
         at System.Text.Json.JsonSerializer.WriteBytesAsObject(Object value, JsonTypeInfo jsonTypeInfo)
         at MeshWeaver.Messaging.Serialization.ObjectPolymorphicConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs:line 435
      …[truncated]
2026-10-05 07:43:37Z memex-portal-deployment-6d8c9665cc-jltnc fail: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Failed to deliver to Hosting/Deployment
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.<AllocateUninitializedArray>g__AllocateNewArrayWorker|77_0[T](Int32 length, Boolean pinned)
         at System.Text.Json.Utf8JsonWriter.WriteStringEscapeValue(ReadOnlySpan`1 value, Int32 firstEscapeIndexVal)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWriteAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsObject(Utf8JsonWriter writer, Object rootValue)
         at System.Text.Json.JsonSerializer.WriteBytesAsObject(Object value, JsonTypeInfo jsonTypeInfo)
         at MeshWeaver.Messaging.Serialization.ObjectPolymorphicConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs:line 435
         at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at Sys…[truncated]
2026-10-05 14:50:27Z memex-portal-deployment-6cfcf78898-gkkdv fail: MeshWeaver.Connection.Orleans.OrleansRoutingService[0]
      Failed to deliver to Admin/content/_Documents/logs-ci-Systemorph-MeshWeaver-111777921171.log
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.AllocateNewArray(IntPtr typeHandlePtr, Int32 length, GC_ALLOC_FLAGS flags, ObjectHandleOnStack ret)
         at System.GC.<AllocateUninitializedArray>g__AllocateNewArrayWorker|77_0[T](Int32 length, Boolean pinned)
         at System.Buffers.SharedArrayPool`1.Rent(Int32 minimumLength)
         at System.Text.Json.Utf8JsonWriter.WriteStringEscapeValue(ReadOnlySpan`1 utf8Value, Int32 firstEscapeIndexVal)
         at System.Text.Json.JsonDocument.WriteComplexElement(Int32 index, Utf8JsonWriter writer)
         at System.Text.Json.JsonDocument.WriteElementTo(Int32 index, Utf8JsonWriter writer)
         at MeshWeaver.Messaging.Serialization.ObjectPolymorphicConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Messaging.Hub/Serialization/ObjectPolymorphicConverter.cs:line 448
         at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at Orleans.Serialization.JsonCodec.Orleans.Serialization.Cloning.IDeepCopier.DeepCopy(Object input, CopyContext context) in /_/src/Orleans.Serialization.SystemTextJson/JsonCodec.cs:line 230
         at OrleansCodeGen.MeshWeaver.Connection.Orleans.Proxy_IRoutingGrain.global::MeshWeaver.Connection.Orleans.IRoutingGrain.RouteMessage(IMessageDelivery arg0) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Connection.Orleans/obj/container/linux-x64/Release/net10.0/Orleans.CodeGenerator/Orleans.CodeGenerator.OrleansSerializationSourceGenerator/MeshWeaver.Connecti…[truncated]

Opened automatically from Admin/_LogIncident/fbd2d04d8cc557b0. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site e763b3f945f48342: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
3h 53m
Merged PRs (30d)
968

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.