StreamProcessor folds long text and tool-call streams in quadratic memory and time

未关闭
  • 哪里不对？
#1,963 0 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@jherr 已经在做这个了。

开始于 2026年11月1日。

评估

这个 Issue 还没有评估数据。

描述

waiting-on: maintainer
Describe the bug

StreamProcessor gets slower and heavier with every delta of one message. Two places cause it, both still in @tanstack/ai 1.0.1 (packages/ai/src/activities/chat/stream/processor.ts).

1. Text deltas hold a copy of the whole message per delta (memory and time).

handleTextMessageContentEvent, line 2343:

const nextText = delta !== '' ? currentText + delta : currentText
const textDelta = nextText.slice(currentText.length)
state.currentSegmentText = nextText
state.totalTextContent += textDelta

In V8, currentText + delta builds a rope. slice on a rope first flattens it into one buffer. For a result of 13 characters or more it then returns a sliced string that points into that buffer, so the delta is not copied out. totalTextContent concatenates those slices, and each one keeps a full flattened copy of the message alive. One message of 10,000 deltas of 16 characters (160 KB of text) holds about 760 MB after a forced GC. 5,000 deltas of 200 characters run Node out of memory at the default 4 GB heap.

textDelta always equals delta here, so the fix is:

state.currentSegmentText = nextText
state.totalTextContent += delta

2. Tool-call arguments are re-parsed in full on every delta (time).

handleToolCallArgsEvent, line 2515:

existingToolCall.arguments += chunk.delta || ''
existingToolCall.parsedArguments = this.jsonParser.parse(existingToolCall.arguments)

The partial parse is O(length of the arguments so far), so a call whose arguments arrive in n deltas costs O(n²). A tool writing a large file streams thousands of TOOL_CALL_ARGS deltas: 4,000 deltas of 100 characters take about 2 s, 10,000 about 12 s. Within the processor only getState() reads parsedArguments before TOOL_CALL_END or the completion path sets the final value. One fix is to parse on read:

// getState()
for (const [id, tc] of state.toolCalls) {
  if (tc.state === 'awaiting-input' || tc.state === 'input-streaming') {
    tc.parsedArguments = this.jsonParser.parse(tc.arguments)
  }
  toolCalls.set(id, tc)
}
Reproduction
import { StreamProcessor } from '@tanstack/ai'
import { setFlagsFromString } from 'node:v8'
import { runInNewContext } from 'node:vm'

setFlagsFromString('--expose-gc')
const gc = runInNewContext('gc')

gc()
const before = process.memoryUsage().heapUsed
const p = new StreamProcessor()
p.processChunk({ type: 'RUN_STARTED', runId: 'r', threadId: 't' })
p.processChunk({ type: 'TEXT_MESSAGE_START', messageId: 'm', role: 'assistant' })
for (let i = 0; i < 10_000; i++)
  p.processChunk({ type: 'TEXT_MESSAGE_CONTENT', messageId: 'm', delta: 'x'.repeat(16) })
gc()
const held = Math.round((process.memoryUsage().heapUsed - before) / 1e6)
console.log(held, 'MB held for', p.getMessages()[0].parts[0].content.length, 'characters')
// 0.61.0, whose code here matches 1.0.1: ~800 MB. With the fix: ~1 MB.

let parsed = 0
const q = new StreamProcessor({ jsonParser: { parse: (s) => ((parsed += s.length), undefined) } })
q.processChunk({ type: 'RUN_STARTED', runId: 'r', threadId: 't' })
q.processChunk({ type: 'TEXT_MESSAGE_START', messageId: 'm', role: 'assistant' })
q.processChunk({ type: 'TOOL_CALL_START', toolCallId: 'c', toolCallName: 'Write', parentMessageId: 'm' })
for (let i = 0; i < 10_000; i++)
  q.processChunk({ type: 'TOOL_CALL_ARGS', toolCallId: 'c', delta: 'x'.repeat(10) })
console.log(parsed, 'characters parsed for 100,000 characters of arguments')
// 0.61.0: 500,050,000. With the fix: 0.
Expected behavior

Folding a message of n deltas takes time and memory linear in its length.

Environment

@tanstack/ai 0.61.0 and 1.0.1, Node 24.21. We run the two fixes above as a local patch on 0.61.0 and can open a PR if that helps.

主要语言
TypeScript
星标
3.3k
派生
378
平均合并
1 天 1 分钟
30 天内合并 PR
208

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

TanStack/ai 的其他 Issue

查看 TanStack/ai 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。