Prematurely-ended provider SSE streams abort agent rounds with no automatic retry (recurrence of the #4726 shape)

Chiusa
  • Qualcosa non va?
#5,890 20 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
backend

Direzione di ricerca

Start with the [ThreadExec] round execution loop in plugins-repo/src/MeshWeaver.AI/ and read the ProviderStreamException classification layer; the issue does not name the loop’s source file or a test. Related source files include StreamStallGuardChatClient.cs and ProviderCallTimingChatClient.cs. Done means transient provider-stream failures no longer silently abort rounds: retry once or post a readable failure to the thread; coordinate with the active discussion and check related issue #4726.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug sev:L
What is failing

A streaming agent round run by MeshWeaver.AI.AgentChatClient (the [ThreadExec] path) was aborted mid-answer when the model endpoint closed the HTTP connection before the SSE response completed. The OpenAI SDK's streaming enumerator surfaced System.Net.Http.HttpIOException (ResponseEnded), the round was lost, and recovery is the manual "resubmit to retry" the exception message itself prescribes — nothing in the round loop retries a failure the classification layer has already declared transient and not the request's fault.

Probable cause

High confidence: upstream endpoint or network fault, not our request. The stack runs entirely through System.Net.Http → System.Net.ServerSentEvents.SseParser → OpenAI.AsyncSseUpdateCollection — no MeshWeaver frame is implicated in the failure itself, and the endpoint terminated the connection mid-response. Our classification layer correctly maps it to a ProviderStreamException with a readable ResponseEnded message. The defect on our side is the missing recovery path: the [ThreadExec] round execution loop lets this self-declared-transient failure kill the round.

Duplicate / related

This is a recurrence of Systemorph/MeshWeaver#4726 (filed from Admin/_LogIncident/5dbcd0029ee06add): identical failure shape — HttpIOException (ResponseEnded) out of the OpenAI SSE enumerator at this log site, thrown while consuming a streaming chat completion. That occurrence escaped unwrapped; the classification layer has since been added, but the missing-retry defect is unchanged and now has a second confirmed occurrence. A human can likely close this as a duplicate of #4726 in seconds — or fold it in as evidence the fault is recurring and the fix (#4726's own proposal) still outstanding.

It is also the fourth distinct provider-stream failure shape at this exact log site within ~10 days, alongside #5819 (upstream finish_reason: "error"), #5859 (30-minute mid-stream stall guard abort) and #5880 (5-minute first-data guard abort). All four share one underlying gap — provider-stream failures in the [ThreadExec] round loop trigger no bounded retry and post no readable failure to the thread — so one catch site fixes them all. Notably, nearly every occurrence across all four shapes hits the review-pull-request-systemorph-* automated PR-review threads on the memex control instance (as this one does); that workload looks disproportionately prone to provider stream faults — worth checking whether its prompts or model route make it so.

Impact

Low: a single aborted round of an internal automation (automated PR review), no data loss, workaround is resubmitting the round. Worth tracking because it is the second confirmed occurrence of this shape and because this log site is accumulating distinct failure shapes against the same missing-recovery gap — if the four families keep recurring, the aggregate deserves a re-rule.

Where to look
  • The [ThreadExec] round execution loop behind the MeshWeaver.AI.AgentChatClient log category, in plugins-repo/src/MeshWeaver.AI/ — catch ProviderStreamException and either retry the round once (the classification already declares the failure transient and the request valid) or post a readable failure to the thread instead of a bare ERROR. Same catch site proposed in #2233, #4726, #5819, #5859 and #5880 — one handler covers all four shapes.
  • The ProviderStreamException classification layer that maps HttpIOException (ResponseEnded) to the "response stream ended before the response did" message — working as intended; the natural place to attach retry semantics.
  • Known source files at this site (from the sibling stacks, not this one): plugins-repo/src/MeshWeaver.AI/StreamStallGuardChatClient.cs, plugins-repo/src/MeshWeaver.AI/ProviderCallTimingChatClient.cs.
  • Sibling incident nodes: Admin/_LogIncident/5dbcd0029ee06add (#4726, prior occurrence of this shape), Admin/_LogIncident/94860cb8c742d627 (#5819), Admin/_LogIncident/8bfa7322a2e2815c (#5859), Admin/_LogIncident/10a0c1a2ce0a446f (#5880).
Evidence
Fingerprint 2ab9a57b70c6c001
Category MeshWeaver.AI.AgentChatClient
Severity Error
Exception MeshWeaver.AI.ProviderStreamException
Top frame OpenAI.AsyncSseUpdateCollection1.AsyncSseUpdateEnumerator1.System.Collections.Generic.IAsyncEnumerator<U>.MoveNextAsync()
Namespace memex
Pods memex-portal-deployment-5dc5995596-d9qj7
Occurrences 1
First seen 2026-09-29 09:11:40Z
Last seen 2026-09-29 09:11:40Z
Routing not determined — no configured route matches the category MeshWeaver.AI.AgentChatClient. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-09-29 09:11:40Z memex-portal-deployment-5dc5995596-d9qj7 fail: MeshWeaver.AI.AgentChatClient[0]
      [ThreadExec] ERROR: 09:11:40.477 threadPath=Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-60bf
      MeshWeaver.AI.ProviderStreamException: The model endpoint's response stream ended before the response did (ResponseEnded): the connection closed mid-answer, so the round was aborted. This is a fault at the upstream endpoint or the network in front of it, not a problem with the request. Resubmit to retry.
       ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
         at System.Net.Http.HttpConnection.FillAsync(Boolean async)
         at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.ReadAsyncCore(Memory`1 buffer, CancellationToken cancellationToken)
         at System.Net.ServerSentEvents.SseParser`1.FillLineBufferAsync(CancellationToken cancellationToken)
         at System.Net.ServerSentEvents.SseParser`1.EnumerateAsync(CancellationToken cancellationToken)+MoveNext()
         at System.Net.ServerSentEvents.SseParser`1.EnumerateAsync(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at OpenAI.AsyncSseUpdateCollection`1.AsyncSseUpdateEnumerator`1.System.Collections.Generic.IAsyncEnumerator<U>.MoveNextAsync()
         at OpenAI.AsyncSseUpdateCollection`1.GetValuesFromPageAsync(ClientResult page)+MoveNext()
         at OpenAI.AsyncSseUpdateCollection`1.GetValuesFromPageAsync(ClientResult page)+MoveNext()
         at OpenAI.AsyncSseUpdateCollection`1.GetValuesFromPageAsync(ClientResult page)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at System.ClientModel.AsyncCollectionResult`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
         at System.ClientModel.AsyncCollectionResult`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
         at System.ClientModel.AsyncCollectionResult`1.GetAsyncEnumerator(Cancellation…[truncated]

Opened automatically from Admin/_LogIncident/2ab9a57b70c6c001. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 1ae89d81a77e5be6: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Lingua principale
C#
Stelle
12
Fork
5
Merge medio
5h 7m
PR unite (30g)
981

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Systemorph/MeshWeaver

Tutte le issue di Systemorph/MeshWeaver

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.