feat(ai): speechSession() for streaming text-to-speech over a live socket

Aperta
  • Qualcosa non va?
#2,046 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@AlemTuzlak ci sta già lavorando.

Dal 5/11/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

enhancement waiting-on: maintainer

Problem

generateSpeech() returns a whole clip. The stream: true option only wraps the finished clip in a StreamChunk iterable; the ElevenLabs adapter still buffers the full response (packages/ai-elevenlabs/src/adapters/speech.ts:200,260). A voice agent waits 1 to 2 s before the first byte of audio plays, so a phone call feels like an IVR, not a conversation.

#1944 added transcriptionSession() for audio in and deferred audio out to a separate issue. This is that issue. The call-centre example built on #1944 works around it by splitting the reply into sentences and calling generateSpeech() per sentence. That gets first audio to about 500 ms but still pays a round trip per sentence and can't stream a long sentence.

Proposal

A speechSession() activity that mirrors transcriptionSession(): text in over a provider WebSocket, audio chunks out as the provider makes them.

import { speechSession } from '@tanstack/ai'
import { elevenlabsLiveSpeech } from '@tanstack/ai-elevenlabs'

const session = await speechSession({
  adapter: elevenlabsLiveSpeech({ voice: 'rachel' }),
  audio: { encoding: 'mulaw', sampleRate: 8000 }, // what the carrier wants
})

for await (const chunk of chat({ ... })) {
  if (chunk.type === 'TEXT_MESSAGE_CONTENT') session.send(chunk.delta)
}
session.flush() // end of turn

for await (const event of session) {
  // { type: 'audio', data: Uint8Array }  play or forward to the carrier
  // { type: 'done' }                      the provider finished the flushed text
  // { type: 'error', error }
}

await session.close()

Barge-in: session.cancel() drops queued text and in-flight audio so the app can send Twilio clear and start a new turn.

Goals
  • Server only, same runtimes as transcriptionSession().
  • Same adapter shape: open, send text, receive events, close. Reuse openLiveSocket from @tanstack/ai/adapter-internals.
  • Telephony formats out of the box: μ-law and A-law at 8 kHz, PCM at 16/24/48 kHz. Adapters declare what they emit; the shared createAudioConverter resamples and encodes when the provider can't.
  • Interim text is cheap. Sending partial sentences must not cost a request per token.
Adapters
Provider Package Socket API
ElevenLabs @tanstack/ai-elevenlabs wss://api.elevenlabs.io/v1/text-to-speech/{voice}/stream-input
OpenAI realtime @tanstack/ai-openai realtime session, text-only input
Deepgram Aura @tanstack/ai-deepgram wss://api.deepgram.com/v1/speak
Cartesia new or community wss://api.cartesia.ai/tts/websocket

Check live-streaming support and ids before building.

Out of scope
  • A browser hook. The browser already has useRealtimeChat.
  • Carrier helpers. Document the pattern, as #1944 did.
  • A full turn-taking loop. The call-centre example shows one.

Follow-up to

#1944, #1964.

Lingua principale
TypeScript
Stelle
3.3k
Fork
378
Merge medio
1g 2h
PR unite (30g)
254

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di TanStack/ai

Tutte le issue di TanStack/ai

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.