Stream write fails with HTTP 409 "inconsistent sequencing state" with concurrent writers (5.5.0, HTTP transport)

Open
#5,418 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
15/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
nodejs, typescript

Research direction

The failure comes from the HTTP stream write path: a PUT to /api/v2/runs/<runId>/stream/<streamName> that returns 409 and is wrapped as a StreamError. Start there, then reproduce two concurrent getWritable() writers on the HTTP transport to see whether the conflict is reachable from client code. Done means either the concurrent-write conflict no longer occurs, or the 409 is classified as a non-retryable dropped write. The issue suggests the server side may be at fault, so this may end in a server change.

Written by the indexing model from the issue text.

Description

Summary

On Vercel, a step that writes to a run's default stream intermittently fails with HTTP 409 and the server message Redis stream "strm_<runId>_user" has inconsistent sequencing state. The failure is deterministic once it starts: every retry of the step gets the same 409, so the step exhausts its retries and the run fails, even though the user-facing work in a parallel step had already succeeded.

Environment

  • workflow / @workflow/core / @workflow/world-vercel 5.5.0
  • Vercel world, Fluid compute, Node.js runtime
  • Streams pinned to the HTTP transport: WORKFLOW_STREAMS_TRANSPORT=http and WORKFLOW_EVENTS_TRANSPORT=http

What happens

  • Two steps of the same run write to the run's default stream at the same time: one writes model output chunks (getWritable() in a "use step" function, lock released in finally), the other writes small progress parts. The streaming docs say concurrent writers are allowed.
  • One write gets:
StreamError: Stream write failed: HTTP 409 (PUT https://vercel-workflow.com/api/v2/runs/<runId>/stream/strm_<runId>_user)
{"success":false,"error":"conflict","message":"Redis stream \"strm_<runId>_user\" has inconsistent sequencing state"}
  • Both writer steps then fail the same way on each retry (Max retries reached, bubbling error to parent workflow), and the run ends with FatalError / USER_ERROR wrapping the StreamError (STREAM_ERROR).
  • In our case the run was still executing on an older deployment while a newer one had been promoted; we have not confirmed whether that matters.

Expected

  • Concurrent writers to the same stream should not put the stream into an inconsistent state, or
  • if the state is unrecoverable, the error should be classified so callers can treat it as a dropped write rather than retrying an append that can never succeed.

Notes

  • On the HTTP transport the client sends no sequence or writer id, so the inconsistent state does not seem reachable from client behaviour; it looks server-side.
  • Possibly related: #4306 (acknowledged chunks missing under concurrent load), #5383 (stream socket closed mid-run on 5.4.0). Separately, #4414 means a step that catches this error still fails after its body returns, which turns a dropped progress write into a failed run.
  • Happy to provide run ids privately if useful.
Dominant language
TypeScript
Stars
2.5k
Forks
401
Avg merge
1d 21h
Merged PRs (30d)
201

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from vercel/workflow

All issues in vercel/workflow

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.