Streams cancel the response body at [DONE], so keep-alive connections are never reused (regression in 7.5.0)

Aperta
  • Qualcosa non va?
#3,050 1 commento 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
54/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
node.js, typescript
Ambito
backend, networking

Direzione di ricerca

The report identifies the stream iterator’s handling of data: [DONE] and response body cleanup, and links regression #2317. Read that change and inspect the stream cleanup path; reproduce with the supplied local HTTP server, which delays ending the chunked response after [DONE]. Done means streamed requests can reuse keep-alive connections in the normal case while still terminating when the response body remains open.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
Confirm this is a Node library issue and not an underlying OpenAI API issue
  • This is an issue with the Node library
Describe the bug

Since v7.5.0 (#2317, "finish SSE streams immediately after [DONE]"), every streamed request uses a new TCP/TLS connection. Before, one keep-alive connection was reused.

When the iterator sees data: [DONE], it breaks out of the loop, and the cleanup cancels the response body (reader.cancel()). For a well-behaved server, the end of the chunked HTTP response arrives a few milliseconds after [DONE], but the SDK has not read it yet. undici cannot return a connection with unread bytes to the pool, so it destroys the socket. The next request has to do a new TCP + TLS handshake.

Side effects we observed with Node 24 (bundled undici):

  • one extra handshake per streamed call: ~80 ms from a home connection, ~10–20 ms from a datacenter, against OpenRouter;
  • one AbortError per stream, captured by APM agents (elastic-apm) even though the SDK swallows it;
  • 2 connections per stream: after the abort, undici re-queues the request and opens a new socket before it checks request.aborted, then closes it right away.

I understand the goal of #2317: not hanging when a server or proxy keeps the response open after [DONE]. A possible fix that keeps that guarantee: after [DONE], keep reading (and discarding) the rest of the body for a short time or size budget (e.g. 1 s / 64 KB). Only cancel if the body has not ended by then. Then the connection goes back to the pool in the normal case.

As far as I can see, no existing issue covers this.

To Reproduce
  1. Save the snippet below as repro.mjs in an empty ESM project ("type": "module").
  2. npm i [email protected] && node repro.mjs → 10 sequential streams -> 20 TCP connections
  3. npm i [email protected] && node repro.mjs → 10 sequential streams -> 1 TCP connections

Results:

openai connections for 10 streams
6.49.0 1
7.4.0 1
7.5.0 20
7.44.0 20
Code snippets
import http from 'node:http';
import OpenAI from 'openai';
import { VERSION } from 'openai/version';

// Minimal OpenAI-compatible SSE server: a few chunks, `data: [DONE]`,
// then a normal end of the chunked response a few ms later.
let connections = 0;
const server = http.createServer((req, res) => {
  req.resume();
  req.on('end', () => {
    res.writeHead(200, { 'content-type': 'text/event-stream' });
    for (const word of ['Hello', ' world']) {
      res.write(`data: ${JSON.stringify({ id: 'x', object: 'chat.completion.chunk', created: 0, model: 'm', choices: [{ index: 0, delta: { content: word }, finish_reason: null }] })}\n\n`);
    }
    res.write('data: [DONE]\n\n');
    setTimeout(() => res.end(), 5);
  });
});
server.on('connection', () => connections++);
await new Promise(r => server.listen(0, r));

const client = new OpenAI({ apiKey: 'test', baseURL: `http://127.0.0.1:${server.address().port}/v1`, maxRetries: 0 });
const calls = 10;
for (let i = 0; i < calls; i++) {
  const stream = await client.chat.completions.create({ model: 'm', stream: true, messages: [{ role: 'user', content: 'hi' }] });
  for await (const _ of stream) {}
  await new Promise(r => setTimeout(r, 50));
}
console.log(`openai ${VERSION} (node ${process.version}): ${calls} sequential streams -> ${connections} TCP connections`);
server.close();
OS

macOS

Node version

v24.18.0

Library version

openai v7.44.0 (regression since v7.5.0)

Lingua principale
TypeScript
Stelle
11.3k
Fork
1.6k
Merge medio
12h 45m
PR unite (30g)
201

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di openai/openai-node

Tutte le issue di openai/openai-node

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.