Stream WebSocket closed before reply fails a durable run mid-stream (doStreamStep, workflow 5.4.0)

Open
#5,383 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active

Research direction

Start with dist/ws-stream-session.js in @workflow/[email protected], especially handleSocketClose, failUnknown, and the chunkSeq handling; the issue gives the observed failure and a unit-test reproduction path using VercelStreamWriteSession. First inspect those paths and the existing session tests, then reproduce a close while a write is pending. Done means the behavior is safely recoverable or its failure is exposed with a documented error type; the issue leaves this design decision open.

Written by the indexing model from the issue text.

Description

Summary

A durable AI chat turn failed in production because the stream WebSocket closed while a chunk write was waiting for its reply. The step fails with stream WebSocket closed before reply. The session is poisoned by design and the step has no retries, so the whole run fails, even though the model call itself was healthy.

I'm filing this to (1) report the failure with exact evidence, (2) ask whether an in-flight write can be made safely recoverable, and (3) ask what could make the server or network close the socket mid-stream. I could not reproduce it on demand: it happened once in roughly 100 runs on a deployment that had used the WebSocket stream transport for about a month. Everything I could determine is below.

Environment

workflow 5.4.0
@workflow/world-vercel 5.4.0
@workflow/serde 5.0.0
@ai-sdk/workflow 2.0.98 (carries a small local patch: client tool durations and no getWorkflowMetadata import; it does not change the stream or closeStream code paths)
ai 7.0.166
next 16.4.2, App Router, withWorkflow()
Node.js 24.22.1
Platform Vercel Functions, production, region iad1
Transport settings defaults: WORKFLOW_STREAMS_TRANSPORT and WORKFLOW_EVENTS_TRANSPORT unset, so the WebSocket stream session is used

Workflow shape: one workflow per chat turn. It calls WorkflowAgent.stream() from @ai-sdk/workflow in a loop (model call, tool calls, model call, ...), and writes UI message chunks to the run's writable (the ..._user stream) so a browser can read them live. Each model call is a doStreamStep, which @ai-sdk/workflow runs with maxRetries = 0 (it must not repeat once output has been written).

What happened

Run A (the failure this issue is about). All times UTC, 2026-11-11.

workflow run id: wrun_41M75XRZ6R0GXVHK7CJC1CZEAY   (deployment id available on request)
18:26:27.6  run started
18:26:29 ... 18:33:22   many earlier doStreamStep and tool steps, all completed normally
18:33:30.332  step_started   doStreamStep           <- the failing step
18:33:36.641  step_failed    doStreamStep           (6.3 s in)
                error: "stream WebSocket closed before reply"
                stack: W.handleSocketClose (@workflow/world-vercel ws-stream-session)
18:33:37.920  checkpointTurnStep completes
18:33:38.459  closeStream (from @ai-sdk/workflow) started, step_retrying at 18:33:38.886, completed 18:33:40.743
18:33:41.7    a post-failure reporting step fails, then the error-reporting steps run
18:33:43.033  run_failed (errorCode USER_ERROR, cause "stream WebSocket closed before reply", fatal)

Observations:

  1. The failing step ran about 6 seconds, a normal length for these steps, so this is not a step-duration or function-duration limit. The run as a whole had been alive for about 7 minutes across many separate step invocations.
  2. The model response was healthy. The failure is the stream write session, not the model provider.
  3. closeStream for the same stream needed a retry on this run, so the stream session was already unhealthy when the turn tried to finish.
  4. In the same minute, other runs on the same deployment (ETL workflows and another chat run) completed normally, so this was not a deployment-wide outage that I can see.
  5. I found no other process-level error (OOM, "Task timed out", crash) in the function logs I could retrieve for that window.

Run B (possibly the same family, a closeStream failure, 15 hours earlier; included because it hits the same session code):

workflow run id: wrun_41M74A010T0GTKJG5BYX37R2FC
Step "closeStream" failed after 3 retries: Stream write failed: HTTP 409
  PUT .../api/v2/runs/<run>/stream/strm_<run>_user
  {"success":false,"error":"conflict","message":"Stream \"strm_<run>_user\" is already completed"}

I have not shown that A and B share a cause. Please treat B as a separate data point. Issue #1949 describes a different 409 ("Run is already completed" on telemetry writes after an early writable.close()).

Frequency: A is the only "stream WebSocket closed before reply" in 90 days of error tracking for this app. The last 100 chat runs in production contained one such failure.

Why it fails (from reading @workflow/[email protected] source)

In dist/ws-stream-session.js:

  • The session comment says an unacknowledged frame "has an unknown outcome and poisons the session rather than being replayed over HTTP or another socket."
  • handleSocketClose(code, socket) returns early for modes closed, http, poisoned, or closeAcknowledged. Otherwise, if this.pending is set, it calls this.failUnknown(new Error('stream WebSocket closed before reply')) (about line 627).
  • If no write is pending, the session reconnects with the same writer identity (idle reconnect, STREAM_WS_RECONNECT_BUDGET_MS = 250), or moves to HTTP after repeated idle reconnects.
  • An unanswered request also poisons the session after getRequestTimeoutMs() ("timed out with no reply"), and a draining session poisons if the drain grace expires with a request pending.
  • failUnknown marks the writer poisoned and rejects the pending write. The step then fails with a FatalError, and the doStreamStep in @ai-sdk/workflow has no retries, so the run fails.

So any socket close that lands while exactly one chunk is awaiting its reply fails the turn. Chunks are small and frequent in an LLM stream, so there is nearly always a pending frame during an active stream. The reconnect-while-idle path handles the other case.

Expected behavior

A transient socket close during an otherwise healthy stream should not fail a durable run. Either the write should be recoverable, or the failure should be one a caller can retry safely.

Questions

  1. What can cause the server or network to close the stream WebSocket about 6 seconds into a step on a healthy deployment (server drain, rolling restart of workflow-server, idle or lifetime limits, platform-side connection recycling)? Is there a server-side log for the stream session of run A at 18:33:36 UTC that shows the close code?
  2. The comment says unknown outcome rules out replay. Could chunk writes carry a writer-local sequence so a reconnect can ask the server for the last acknowledged sequence (or send idempotently)? The code already has a chunkSeq and a writer id, which suggests the information exists.
  3. If not, could the failure be surfaced as a distinct, documented error type so @ai-sdk/workflow and applications can decide whether to retry the step, restart the stream, or fall back to HTTP for the rest of the run?
  4. closeStream retried on run A and exhausted retries on run B with a 409 "already completed". Is close after a poisoned or reconnected WebSocket session expected to be safe to repeat?

Workaround

Setting WORKFLOW_STREAMS_TRANSPORT=http opts out of the WebSocket stream session. HTTP appends use retryStreamAppend, which the WebSocket path deliberately lacks. We applied this in production after this failure. I don't yet have data on whether it removes the failure or on its latency cost.

How to try to reproduce

I have no on-demand reproduction, only the conditions. A reproduction attempt on your side:

  1. Create a Next.js app with withWorkflow() on [email protected] (Vercel, production-like, WORKFLOW_STREAMS_TRANSPORT unset).
  2. Write a workflow that calls WorkflowAgent.stream() from @ai-sdk/[email protected] with [email protected] and a model that produces a multi-second streamed response, and write the UI message stream to getWritable(). A read of that stream from a browser or workflow inspect streams consumer is not required for the failure.
  3. While a doStreamStep is actively writing chunks, make the stream WebSocket close with an unacknowledged frame. Any of: (a) restart or drain the workflow-server side of the connection, (b) drop the TCP connection or send a close frame with code 1001/1006 from a proxy placed in front of the stream endpoint, (c) in a unit test of VercelStreamWriteSession, set pending by issuing a write and then invoke the socket close handler before a reply arrives.
  4. Expected with the current code: the step fails with stream WebSocket closed before reply, the doStreamStep is not retried, and the run ends failed with errorCode: USER_ERROR.
  5. For the closeStream 409 (run B), I don't have a reproduction. It occurred after the final model step, after a retried closeStream.

If it helps, I can share the full decrypted error output and the step and event listing for runs A and B, the deployment id, and the matching production log lines privately.

Dominant language
TypeScript
Stars
2.5k
Forks
401
Avg merge
1d 21h
Merged PRs (30d)
201

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from vercel/workflow

All issues in vercel/workflow

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.