feat(ai): speechSession() for streaming text-to-speech over a live socket

未关闭
  • 哪里不对？
#2,046 0 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@AlemTuzlak 已经在做这个了。

开始于 2026年11月5日。

评估

这个 Issue 还没有评估数据。

描述

enhancement waiting-on: maintainer

Problem

generateSpeech() returns a whole clip. The stream: true option only wraps the finished clip in a StreamChunk iterable; the ElevenLabs adapter still buffers the full response (packages/ai-elevenlabs/src/adapters/speech.ts:200,260). A voice agent waits 1 to 2 s before the first byte of audio plays, so a phone call feels like an IVR, not a conversation.

#1944 added transcriptionSession() for audio in and deferred audio out to a separate issue. This is that issue. The call-centre example built on #1944 works around it by splitting the reply into sentences and calling generateSpeech() per sentence. That gets first audio to about 500 ms but still pays a round trip per sentence and can't stream a long sentence.

Proposal

A speechSession() activity that mirrors transcriptionSession(): text in over a provider WebSocket, audio chunks out as the provider makes them.

import { speechSession } from '@tanstack/ai'
import { elevenlabsLiveSpeech } from '@tanstack/ai-elevenlabs'

const session = await speechSession({
  adapter: elevenlabsLiveSpeech({ voice: 'rachel' }),
  audio: { encoding: 'mulaw', sampleRate: 8000 }, // what the carrier wants
})

for await (const chunk of chat({ ... })) {
  if (chunk.type === 'TEXT_MESSAGE_CONTENT') session.send(chunk.delta)
}
session.flush() // end of turn

for await (const event of session) {
  // { type: 'audio', data: Uint8Array }  play or forward to the carrier
  // { type: 'done' }                      the provider finished the flushed text
  // { type: 'error', error }
}

await session.close()

Barge-in: session.cancel() drops queued text and in-flight audio so the app can send Twilio clear and start a new turn.

Goals
  • Server only, same runtimes as transcriptionSession().
  • Same adapter shape: open, send text, receive events, close. Reuse openLiveSocket from @tanstack/ai/adapter-internals.
  • Telephony formats out of the box: μ-law and A-law at 8 kHz, PCM at 16/24/48 kHz. Adapters declare what they emit; the shared createAudioConverter resamples and encodes when the provider can't.
  • Interim text is cheap. Sending partial sentences must not cost a request per token.
Adapters
Provider Package Socket API
ElevenLabs @tanstack/ai-elevenlabs wss://api.elevenlabs.io/v1/text-to-speech/{voice}/stream-input
OpenAI realtime @tanstack/ai-openai realtime session, text-only input
Deepgram Aura @tanstack/ai-deepgram wss://api.deepgram.com/v1/speak
Cartesia new or community wss://api.cartesia.ai/tts/websocket

Check live-streaming support and ids before building.

Out of scope
  • A browser hook. The browser already has useRealtimeChat.
  • Carrier helpers. Document the pattern, as #1944 did.
  • A full turn-taking loop. The call-centre example shows one.

Follow-up to

#1944, #1964.

主要语言
TypeScript
星标
3.3k
派生
378
平均合并
1 天 2 小时
30 天内合并 PR
254

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

TanStack/ai 的其他 Issue

查看 TanStack/ai 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。