Optional Streamfold example for incremental structured-output previews

Open
  • Is something wrong?
#3,080 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
node.js, typescript
Domain
developer-experience

Research direction

Read the existing incremental structured-output helpers and examples, then compare their event handling and schema validation path with the proposed Anthropic example at PR 1291. The issue gives no target file or test entry point, so first locate those in the repository. Done means an opt-in example consumes Responses text deltas with Streamfold, exposes partial values or patches, and keeps final schema validation in the SDK, with focused conformance tests.

Written by the indexing model from the issue text.

Description

Proposal

Would an optional example for incremental structured-output previews using Streamfold be useful here? I am following the contributor policy directing non-collaborators to issues. Streamfold is maintained under assistant-ui.

The proposed example would consume raw Responses text deltas, feed each new chunk to Streamfold, and expose partial values or patches for a UI. Final structured output would still go through the existing schema validation path. Streamfold would be an example-only dependency, with no change to the SDK default parser, public event semantics, or runtime dependencies.

Evidence and limits

I compared the vendored partialParse from openai 7.45.1 with streamfold 0.1.13 on synthetic arrays, checking every partial and final value for equality. Results below are milliseconds per complete replay, median / p95, on Apple M1 and Node 24.22.1. There are five warmups and twenty measured trials, alternating execution order. Streamfold uses immutable snapshots.

UTF-8 bytes Chunks Reparse median / p95 Streamfold median / p95
579 10 0.151 / 0.538 0.181 / 0.642
9,391 130 15.025 / 15.408 1.709 / 2.510
38,191 514 229.357 / 236.258 6.815 / 7.190

The small input is slower with Streamfold. The larger inputs show a potential opportunity when an application reparses every growing prefix. These are parser microbenchmarks, not measurements of the complete SDK helpers, network/model latency, rendering, or first-use WASM compilation. The SDK's own parse scheduling and limits are outside this benchmark. Each delta contains one complete array record, so these results do not establish performance for arbitrary token boundaries. Partial-string semantics also differ between the parsers; this is not evidence for replacing the default parser transparently.

A runnable Anthropic example following the same opt-in approach is proposed in https://github.com/anthropics/anthropic-sdk-typescript/pull/1291. It keeps final schema validation in the SDK and covers partial snapshots, malformed output, cancellation, and multiple text blocks through the real SDK decoder.

Reproduction

In a temporary ESM project, install [email protected] and [email protected], save the following as benchmark.mjs, and run node benchmark.mjs. It makes no API requests.

import assert from "node:assert/strict";
import { cpus } from "node:os";
import { performance } from "node:perf_hooks";
const { IncrementalJsonScanner } = await import(process.argv[2] ?? "streamfold");
const { partialParse } = await import("openai/_vendor/partial-json-parser/parser");

const results = [];
for (const count of [8, 128, 512]) {
  const rows = Array.from({ length: count }, (_, id) => ({ id, title: `Task ${id}: café`, done: false, tags: ["work", "review"] }));
  const chunks = ['{"items":[', ...rows.map((row, index) => `${index ? "," : ""}${JSON.stringify(row)}`), "]}"];
  const text = chunks.join("");
  const scanner = new IncrementalJsonScanner({ snapshots: "immutable" });
  let prefix = "";
  for (const chunk of chunks) {
    prefix += chunk;
    assert.deepEqual(scanner.push(chunk).partialValue, partialParse(prefix));
  }
  assert.deepEqual(scanner.finish().partialValue, { items: rows });
  scanner.dispose();
  const run = [
    () => {
      let input = "";
      for (const chunk of chunks) { input += chunk; partialParse(input); }
      return JSON.parse(input).items.length;
    },
    () => {
      const input = new IncrementalJsonScanner({ snapshots: "immutable" });
      try {
        for (const chunk of chunks) input.push(chunk);
        return input.finish().partialValue.items.length;
      } finally { input.dispose(); }
    },
  ];
  const samples = [[], []];
  for (let trial = -5; trial < 20; trial++) {
    for (const index of trial % 2 === 0 ? [0, 1] : [1, 0]) {
      const start = performance.now();
      const length = run[index]();
      const elapsed = performance.now() - start;
      assert.equal(length, count);
      if (trial >= 0) samples[index].push(elapsed);
    }
  }
  const stats = samples.map((values) => {
    values.sort((a, b) => a - b);
    return { medianMs: (values[9] + values[10]) / 2, p95Ms: values[18] };
  });
  results.push({ count, bytes: Buffer.byteLength(text), chunks: chunks.length, baseline: stats[0], streamfold: stats[1], ratio: stats[0].medianMs / stats[1].medianMs });
}
console.log(JSON.stringify({ node: process.version, cpu: cpus()[0].model, warmups: 5, trials: 20, snapshots: "immutable", chunks: "one complete array record per delta; partial values checked for equality", results }, null, 2));

If an example is welcome, I can provide the corresponding implementation and focused conformance tests through the contribution route the maintainers prefer.

Dominant language
TypeScript
Stars
11.3k
Forks
1.6k
Avg merge
12h 40m
Merged PRs (30d)
204

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openai/openai-node

All issues in openai/openai-node

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.