Stream WebSocket closed before reply fails a durable run mid-stream (doStreamStep, workflow 5.4.0)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- next.js, node.js, typescript
- Domain
- backend, distributed-systems, networking
Research direction
Start with dist/ws-stream-session.js in @workflow/[email protected], especially handleSocketClose, failUnknown, and the chunkSeq handling; the issue gives the observed failure and a unit-test reproduction path using VercelStreamWriteSession. First inspect those paths and the existing session tests, then reproduce a close while a write is pending. Done means the behavior is safely recoverable or its failure is exposed with a documented error type; the issue leaves this design decision open.
Written by the indexing model from the issue text.
Description
Summary
A durable AI chat turn failed in production because the stream WebSocket closed while a chunk write was waiting for its reply. The step fails with stream WebSocket closed before reply. The session is poisoned by design and the step has no retries, so the whole run fails, even though the model call itself was healthy.
I'm filing this to (1) report the failure with exact evidence, (2) ask whether an in-flight write can be made safely recoverable, and (3) ask what could make the server or network close the socket mid-stream. I could not reproduce it on demand: it happened once in roughly 100 runs on a deployment that had used the WebSocket stream transport for about a month. Everything I could determine is below.
Environment
workflow |
5.4.0 |
@workflow/world-vercel |
5.4.0 |
@workflow/serde |
5.0.0 |
@ai-sdk/workflow |
2.0.98 (carries a small local patch: client tool durations and no getWorkflowMetadata import; it does not change the stream or closeStream code paths) |
ai |
7.0.166 |
next |
16.4.2, App Router, withWorkflow() |
| Node.js | 24.22.1 |
| Platform | Vercel Functions, production, region iad1 |
| Transport settings | defaults: WORKFLOW_STREAMS_TRANSPORT and WORKFLOW_EVENTS_TRANSPORT unset, so the WebSocket stream session is used |
Workflow shape: one workflow per chat turn. It calls WorkflowAgent.stream() from @ai-sdk/workflow in a loop (model call, tool calls, model call, ...), and writes UI message chunks to the run's writable (the ..._user stream) so a browser can read them live. Each model call is a doStreamStep, which @ai-sdk/workflow runs with maxRetries = 0 (it must not repeat once output has been written).
What happened
Run A (the failure this issue is about). All times UTC, 2026-11-11.
workflow run id: wrun_41M75XRZ6R0GXVHK7CJC1CZEAY (deployment id available on request)
18:26:27.6 run started
18:26:29 ... 18:33:22 many earlier doStreamStep and tool steps, all completed normally
18:33:30.332 step_started doStreamStep <- the failing step
18:33:36.641 step_failed doStreamStep (6.3 s in)
error: "stream WebSocket closed before reply"
stack: W.handleSocketClose (@workflow/world-vercel ws-stream-session)
18:33:37.920 checkpointTurnStep completes
18:33:38.459 closeStream (from @ai-sdk/workflow) started, step_retrying at 18:33:38.886, completed 18:33:40.743
18:33:41.7 a post-failure reporting step fails, then the error-reporting steps run
18:33:43.033 run_failed (errorCode USER_ERROR, cause "stream WebSocket closed before reply", fatal)
Observations:
- The failing step ran about 6 seconds, a normal length for these steps, so this is not a step-duration or function-duration limit. The run as a whole had been alive for about 7 minutes across many separate step invocations.
- The model response was healthy. The failure is the stream write session, not the model provider.
closeStreamfor the same stream needed a retry on this run, so the stream session was already unhealthy when the turn tried to finish.- In the same minute, other runs on the same deployment (ETL workflows and another chat run) completed normally, so this was not a deployment-wide outage that I can see.
- I found no other process-level error (OOM, "Task timed out", crash) in the function logs I could retrieve for that window.
Run B (possibly the same family, a closeStream failure, 15 hours earlier; included because it hits the same session code):
workflow run id: wrun_41M74A010T0GTKJG5BYX37R2FC
Step "closeStream" failed after 3 retries: Stream write failed: HTTP 409
PUT .../api/v2/runs/<run>/stream/strm_<run>_user
{"success":false,"error":"conflict","message":"Stream \"strm_<run>_user\" is already completed"}
I have not shown that A and B share a cause. Please treat B as a separate data point. Issue #1949 describes a different 409 ("Run is already completed" on telemetry writes after an early writable.close()).
Frequency: A is the only "stream WebSocket closed before reply" in 90 days of error tracking for this app. The last 100 chat runs in production contained one such failure.
Why it fails (from reading @workflow/[email protected] source)
In dist/ws-stream-session.js:
- The session comment says an unacknowledged frame "has an unknown outcome and poisons the session rather than being replayed over HTTP or another socket."
handleSocketClose(code, socket)returns early for modesclosed,http,poisoned, orcloseAcknowledged. Otherwise, ifthis.pendingis set, it callsthis.failUnknown(new Error('stream WebSocket closed before reply'))(about line 627).- If no write is pending, the session reconnects with the same writer identity (idle reconnect,
STREAM_WS_RECONNECT_BUDGET_MS = 250), or moves to HTTP after repeated idle reconnects. - An unanswered request also poisons the session after
getRequestTimeoutMs()("timed out with no reply"), and adrainingsession poisons if the drain grace expires with a request pending. failUnknownmarks the writerpoisonedand rejects the pending write. The step then fails with aFatalError, and thedoStreamStepin@ai-sdk/workflowhas no retries, so the run fails.
So any socket close that lands while exactly one chunk is awaiting its reply fails the turn. Chunks are small and frequent in an LLM stream, so there is nearly always a pending frame during an active stream. The reconnect-while-idle path handles the other case.
Expected behavior
A transient socket close during an otherwise healthy stream should not fail a durable run. Either the write should be recoverable, or the failure should be one a caller can retry safely.
Questions
- What can cause the server or network to close the stream WebSocket about 6 seconds into a step on a healthy deployment (server drain, rolling restart of workflow-server, idle or lifetime limits, platform-side connection recycling)? Is there a server-side log for the stream session of run A at 18:33:36 UTC that shows the close code?
- The comment says unknown outcome rules out replay. Could chunk writes carry a writer-local sequence so a reconnect can ask the server for the last acknowledged sequence (or send idempotently)? The code already has a
chunkSeqand a writer id, which suggests the information exists. - If not, could the failure be surfaced as a distinct, documented error type so
@ai-sdk/workflowand applications can decide whether to retry the step, restart the stream, or fall back to HTTP for the rest of the run? closeStreamretried on run A and exhausted retries on run B with a 409 "already completed". Iscloseafter a poisoned or reconnected WebSocket session expected to be safe to repeat?
Workaround
Setting WORKFLOW_STREAMS_TRANSPORT=http opts out of the WebSocket stream session. HTTP appends use retryStreamAppend, which the WebSocket path deliberately lacks. We applied this in production after this failure. I don't yet have data on whether it removes the failure or on its latency cost.
How to try to reproduce
I have no on-demand reproduction, only the conditions. A reproduction attempt on your side:
- Create a Next.js app with
withWorkflow()on[email protected](Vercel, production-like,WORKFLOW_STREAMS_TRANSPORTunset). - Write a workflow that calls
WorkflowAgent.stream()from@ai-sdk/[email protected]with[email protected]and a model that produces a multi-second streamed response, and write the UI message stream togetWritable(). A read of that stream from a browser orworkflow inspect streamsconsumer is not required for the failure. - While a
doStreamStepis actively writing chunks, make the stream WebSocket close with an unacknowledged frame. Any of: (a) restart or drain the workflow-server side of the connection, (b) drop the TCP connection or send a close frame with code 1001/1006 from a proxy placed in front of the stream endpoint, (c) in a unit test ofVercelStreamWriteSession, setpendingby issuing a write and then invoke the socketclosehandler before a reply arrives. - Expected with the current code: the step fails with
stream WebSocket closed before reply, thedoStreamStepis not retried, and the run endsfailedwitherrorCode: USER_ERROR. - For the
closeStream409 (run B), I don't have a reproduction. It occurred after the final model step, after a retriedcloseStream.
If it helps, I can share the full decrypted error output and the step and event listing for runs A and B, the deployment id, and the matching production log lines privately.
- Dominant language
- TypeScript
- Stars
- 2.5k
- Forks
- 401
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 201
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from vercel/workflow
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
node-module-esbuild-plugin: getImportedIdentifier only checks the first named import when locating a violationPossibly taken @div-cowboy claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
vercel/workflow#5057 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
code-transform.mdx §ID Generation omits the package-shipped `name@version` ID formPossibly taken @Mohith26 claimed this 113 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 90/100
Maintainers usually reply within 1 day
Similar issues
-
db-d1-sqlite: upsert is aliased to updateOne, so payload.kv.set never inserts new keysPossibly taken @pcbeingused333 claimed this today. OpenBug db: d1-sqlite Payload OSS
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
payloadcms/payload#19766 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
device-management-toolkit/sample-web-ui#3711 ·
Maintainers usually reply within 1 day
-
bug impact:low Team: SecuritySolution Team:Defend Workflows v9.6.0
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
elastic/kibana#306401 · 3 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
agentes-chat/agentes-chat#927 ·
Maintainers usually reply within 1 day
-
Closing a pinned tab jumps to the Tabs section instead of the next open pinned tabPossibly taken @seenimohamed claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ardia-browser/ardia#2769 ·
Maintainers usually reply within 1 day