Stream write fails with HTTP 409 "inconsistent sequencing state" with concurrent writers (5.5.0, HTTP transport)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 15/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- nodejs, typescript
- Domain
- backend, distributed-systems
Research direction
The failure comes from the HTTP stream write path: a PUT to /api/v2/runs/<runId>/stream/<streamName> that returns 409 and is wrapped as a StreamError. Start there, then reproduce two concurrent getWritable() writers on the HTTP transport to see whether the conflict is reachable from client code. Done means either the concurrent-write conflict no longer occurs, or the 409 is classified as a non-retryable dropped write. The issue suggests the server side may be at fault, so this may end in a server change.
Written by the indexing model from the issue text.
Description
Summary
On Vercel, a step that writes to a run's default stream intermittently fails with HTTP 409 and the server message Redis stream "strm_<runId>_user" has inconsistent sequencing state. The failure is deterministic once it starts: every retry of the step gets the same 409, so the step exhausts its retries and the run fails, even though the user-facing work in a parallel step had already succeeded.
Environment
workflow/@workflow/core/@workflow/world-vercel5.5.0- Vercel world, Fluid compute, Node.js runtime
- Streams pinned to the HTTP transport:
WORKFLOW_STREAMS_TRANSPORT=httpandWORKFLOW_EVENTS_TRANSPORT=http
What happens
- Two steps of the same run write to the run's default stream at the same time: one writes model output chunks (
getWritable()in a"use step"function, lock released infinally), the other writes small progress parts. The streaming docs say concurrent writers are allowed. - One write gets:
StreamError: Stream write failed: HTTP 409 (PUT https://vercel-workflow.com/api/v2/runs/<runId>/stream/strm_<runId>_user)
{"success":false,"error":"conflict","message":"Redis stream \"strm_<runId>_user\" has inconsistent sequencing state"}
- Both writer steps then fail the same way on each retry (
Max retries reached, bubbling error to parent workflow), and the run ends withFatalError/USER_ERRORwrapping theStreamError(STREAM_ERROR). - In our case the run was still executing on an older deployment while a newer one had been promoted; we have not confirmed whether that matters.
Expected
- Concurrent writers to the same stream should not put the stream into an inconsistent state, or
- if the state is unrecoverable, the error should be classified so callers can treat it as a dropped write rather than retrying an append that can never succeed.
Notes
- On the HTTP transport the client sends no sequence or writer id, so the inconsistent state does not seem reachable from client behaviour; it looks server-side.
- Possibly related: #4306 (acknowledged chunks missing under concurrent load), #5383 (stream socket closed mid-run on 5.4.0). Separately, #4414 means a step that catches this error still fails after its body returns, which turns a dropped progress write into a failed run.
- Happy to provide run ids privately if useful.
- Dominant language
- TypeScript
- Stars
- 2.5k
- Forks
- 401
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 201
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from vercel/workflow
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
node-module-esbuild-plugin: getImportedIdentifier only checks the first named import when locating a violationPossibly taken @div-cowboy claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
vercel/workflow#5057 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
code-transform.mdx §ID Generation omits the package-shipped `name@version` ID formPossibly taken @Mohith26 claimed this 113 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 90/100
Maintainers usually reply within 1 day
Similar issues
-
db-d1-sqlite: upsert is aliased to updateOne, so payload.kv.set never inserts new keysPossibly taken @pcbeingused333 claimed this today. OpenBug db: d1-sqlite Payload OSS
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
payloadcms/payload#19766 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
device-management-toolkit/sample-web-ui#3711 ·
Maintainers usually reply within 1 day
-
bug impact:low Team: SecuritySolution Team:Defend Workflows v9.6.0
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
elastic/kibana#306401 · 3 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
agentes-chat/agentes-chat#927 ·
Maintainers usually reply within 1 day
-
Closing a pinned tab jumps to the Tabs section instead of the next open pinned tabPossibly taken @seenimohamed claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ardia-browser/ardia#2769 ·
Maintainers usually reply within 1 day