`ChatEngine.handleTextMessageContentEvent` is O(n²) in answer length: `startsWith(before)` copies the whole accumulated text on every chunk

Cerrado
  • ¿Algo no está bien?
#1,951 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@tombeckenham ya está trabajando en esto.

Desde el 30/10/2026.

  • #1962 de @tombeckenham — fusionado

Evaluación

Este issue todavía no se ha evaluado.

Descripción

has-pr waiting-on: maintainer
What happens

handleTextMessageContentEvent in packages/ai/src/activities/chat/index.ts (main @ a7eb584, L2405–L2431) runs this for every TEXT_MESSAGE_CONTENT chunk:

const before = this.accumulatedContent
if (typeof extra.content === 'string' && extra.content !== '') {
  this.accumulatedContent = extra.content
} else {
  this.accumulatedContent += chunk.delta
}
// ...
if (!this.accumulatedContent.startsWith(before)) {
  this.turnParts = null
  return
}
const delta = this.accumulatedContent.slice(before.length)

After +=, V8 doesn't copy the text. It stores the result as a string that points to the two parts. startsWith then forces V8 to copy the whole accumulated answer into one string, on every chunk. So the total cost grows with the square of the answer length, and none of it yields to the event loop.

Reproduction

This uses the same string operations, with no TanStack code involved. Node 22, Apple Silicon:

const N = 195_000
const chunk = 'abcd'

let acc = ''
const t0 = performance.now()
for (let i = 0; i < N / chunk.length; i++) {
  const before = acc
  acc += chunk
  if (!acc.startsWith(before)) {
    throw new Error('unreachable')
  }
  acc.slice(before.length)
}
console.log(performance.now() - t0)
  • With the startsWith, it takes about 10.9 s.
  • With only the +=, it takes about 0 ms.

In a real chat() run, a CPU profile of a stream of about 195k characters in 4-character chunks showed about 15.7 s inside handleStreamChunk. A model that returns a long answer in small chunks blocks the server's event loop for that whole time.

We noticed it in a test that streams a large fake answer four characters at a time. Writing the same bytes as 3-byte CJK characters cut the character count to a quarter and the runtime by more than ten times, about what O(n²) in string length predicts. A model that streams a long answer in small deltas hits the same cost in production.

Affected versions: @tanstack/ai 1.0.0 and 1.0.1, and current main.

Suggested fix

The prefix check only matters when an adapter sends a cumulative content snapshot. When the chunk is a plain delta, the new text starts with before by definition, and the new text is chunk.delta. So the check can move into the snapshot branch:

const extra = chunk as AdapterYieldChunk
const before = this.accumulatedContent
let delta: string
if (typeof extra.content === 'string' && extra.content !== '') {
  this.accumulatedContent = extra.content
  if (this.turnParts && !extra.content.startsWith(before)) {
    // A cumulative `content` snapshot rewrote earlier text; order unknown.
    this.turnParts = null
  }
  delta = extra.content.slice(before.length)
} else {
  this.accumulatedContent += chunk.delta
  delta = chunk.delta
}
this.middlewareCtx.accumulatedContent = this.accumulatedContent
if (!this.turnParts) return
if (delta === '') return
const last = this.turnParts[this.turnParts.length - 1]
if (last && last.type === 'text') {
  last.content += delta
} else {
  this.turnParts.push({
    type: 'text',
    content: delta,
  })
}

With this change, a stream of deltas does constant work per chunk. Snapshot chunks still copy the text, but they are the rare fallback path.

A regression test could stream about 200k characters of 4-character deltas through chat() with a mock adapter, then check that the turn's text part matches the input and that the run takes well under a second. If you'd rather not test timing, assert on the result and treat the benchmark above as the guard.

Lenguaje dominante
TypeScript
Estrellas
3.3k
Forks
378
Merge medio
1 d 2 h
PR fusionados (30 d)
219

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de TanStack/ai

Todos los issues de TanStack/ai

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.