Streams cancel the response body at [DONE], so keep-alive connections are never reused (regression in 7.5.0)

Open
  • Is something wrong?
#3,050 1 comment 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
54/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
node.js, typescript
Domain
backend, networking

Research direction

The report identifies the stream iterator’s handling of data: [DONE] and response body cleanup, and links regression #2317. Read that change and inspect the stream cleanup path; reproduce with the supplied local HTTP server, which delays ending the chunked response after [DONE]. Done means streamed requests can reuse keep-alive connections in the normal case while still terminating when the response body remains open.

Written by the indexing model from the issue text.

Description

bug
Confirm this is a Node library issue and not an underlying OpenAI API issue
  • This is an issue with the Node library
Describe the bug

Since v7.5.0 (#2317, "finish SSE streams immediately after [DONE]"), every streamed request uses a new TCP/TLS connection. Before, one keep-alive connection was reused.

When the iterator sees data: [DONE], it breaks out of the loop, and the cleanup cancels the response body (reader.cancel()). For a well-behaved server, the end of the chunked HTTP response arrives a few milliseconds after [DONE], but the SDK has not read it yet. undici cannot return a connection with unread bytes to the pool, so it destroys the socket. The next request has to do a new TCP + TLS handshake.

Side effects we observed with Node 24 (bundled undici):

  • one extra handshake per streamed call: ~80 ms from a home connection, ~10–20 ms from a datacenter, against OpenRouter;
  • one AbortError per stream, captured by APM agents (elastic-apm) even though the SDK swallows it;
  • 2 connections per stream: after the abort, undici re-queues the request and opens a new socket before it checks request.aborted, then closes it right away.

I understand the goal of #2317: not hanging when a server or proxy keeps the response open after [DONE]. A possible fix that keeps that guarantee: after [DONE], keep reading (and discarding) the rest of the body for a short time or size budget (e.g. 1 s / 64 KB). Only cancel if the body has not ended by then. Then the connection goes back to the pool in the normal case.

As far as I can see, no existing issue covers this.

To Reproduce
  1. Save the snippet below as repro.mjs in an empty ESM project ("type": "module").
  2. npm i [email protected] && node repro.mjs → 10 sequential streams -> 20 TCP connections
  3. npm i [email protected] && node repro.mjs → 10 sequential streams -> 1 TCP connections

Results:

openai connections for 10 streams
6.49.0 1
7.4.0 1
7.5.0 20
7.44.0 20
Code snippets
import http from 'node:http';
import OpenAI from 'openai';
import { VERSION } from 'openai/version';

// Minimal OpenAI-compatible SSE server: a few chunks, `data: [DONE]`,
// then a normal end of the chunked response a few ms later.
let connections = 0;
const server = http.createServer((req, res) => {
  req.resume();
  req.on('end', () => {
    res.writeHead(200, { 'content-type': 'text/event-stream' });
    for (const word of ['Hello', ' world']) {
      res.write(`data: ${JSON.stringify({ id: 'x', object: 'chat.completion.chunk', created: 0, model: 'm', choices: [{ index: 0, delta: { content: word }, finish_reason: null }] })}\n\n`);
    }
    res.write('data: [DONE]\n\n');
    setTimeout(() => res.end(), 5);
  });
});
server.on('connection', () => connections++);
await new Promise(r => server.listen(0, r));

const client = new OpenAI({ apiKey: 'test', baseURL: `http://127.0.0.1:${server.address().port}/v1`, maxRetries: 0 });
const calls = 10;
for (let i = 0; i < calls; i++) {
  const stream = await client.chat.completions.create({ model: 'm', stream: true, messages: [{ role: 'user', content: 'hi' }] });
  for await (const _ of stream) {}
  await new Promise(r => setTimeout(r, 50));
}
console.log(`openai ${VERSION} (node ${process.version}): ${calls} sequential streams -> ${connections} TCP connections`);
server.close();
OS

macOS

Node version

v24.18.0

Library version

openai v7.44.0 (regression since v7.5.0)

Dominant language
TypeScript
Stars
11.3k
Forks
1.6k
Avg merge
12h 40m
Merged PRs (30d)
204

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openai/openai-node

All issues in openai/openai-node

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.