feat: live speech-to-text sessions for voice and telephony agents

Ouverte
  • Quelque chose ne va pas ?
#1,944 0 commentaires 0 réactions 1 personne assignée Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@AlemTuzlak y travaille déjà.

Depuis le 29/10/2026.

  • #1964 par @tombeckenham — ouverte

Évaluation

Cette issue n'a pas encore été évaluée.

Description

waiting-on: maintainer

Problem

TanStack AI can't turn live audio into text on the server. That blocks the whole voice and telephony agent space: phone agents, call centres, IVR replacement, meeting bots, and voice agents behind an SFU.

  • generateTranscription() only takes a finished file.
  • Realtime voice (useRealtimeChat) connects the browser straight to the provider. The connection reads the microphone itself (startAudioCapture), so a server can't feed it audio from a phone call.

A server-owned voice agent needs to feed audio as it arrives, and get text back while the caller talks. The audio comes from Twilio Media Streams, Telnyx, Plivo, Vonage, LiveKit, an SFU, or a browser WebSocket. Today teams stitch this together from vendor SDKs, then hand the text to chat().

This came up from a user who runs a voice agent on Cloudflare agents with withVoice. Their reply text already comes from chat(). Speech-to-text is the piece TanStack can't provide. See Transcriber in agents/voice for one pipeline's view of the shape.

Proposal

A new activity, a live transcription session, with one provider-neutral adapter contract.

import { transcriptionSession } from '@tanstack/ai'
import { deepgramLiveTranscription } from '@tanstack/ai-deepgram'

const session = await transcriptionSession({
  adapter: deepgramLiveTranscription({ model: 'nova-3' }),
  audio: { encoding: 'mulaw', sampleRate: 8000, channels: 1 }, // a Twilio call
  language: 'en',
})

twilioSocket.on('media', (m) => session.send(m.payload)) // raw audio chunks

for await (const event of session) {
  // { type: 'interim', text }  text that may still change
  // { type: 'final', text }    stable text for a segment
  // { type: 'turn-end' }       the provider thinks the caller finished talking
  // { type: 'error', error }
}

await session.close()

Swap the adapter to change provider. Nothing else changes.

Goals
  • Runs on the server: Node, Bun, Workers and Durable Objects. No browser APIs.
  • One small adapter contract (open, send audio, receive events, close), so providers are cheap to add, including community adapters.
  • Telephony audio is first-class: G.711 μ-law and A-law at 8 kHz, plus linear PCM at 8, 16, 24 and 48 kHz. Each adapter declares what it accepts. A shared helper converts formats when the provider can't take the input as is.
  • Optional turn-end signal, when the provider has endpointing.
  • Barge-in support: the caller's speech during agent playback must reach the app fast, as interim events.
  • Small enough to wrap for other voice pipelines, for example an agents/voice Transcriber, in a few lines.
Providers

The first PR ships at least two providers from different vendors. That proves the contract isn't shaped around one API.

Provider Package Notes
Deepgram (Nova-3, Flux) new @tanstack/ai-deepgram Most common STT in telephony stacks. Flux has turn detection
AssemblyAI (Universal-Streaming) new @tanstack/ai-assemblyai Turn detection built in
ElevenLabs Scribe realtime @tanstack/ai-elevenlabs Package exists
OpenAI realtime transcription @tanstack/ai-openai Package exists
Workers AI (Deepgram models over the AI binding) @tanstack/ai-cloudflare Package exists. The user who asked
Gladia, Speechmatics, Soniox, Google, Azure community or later

I haven't confirmed live-streaming support or exact model ids for each row. Check before you build.

Out of scope for the first PR

  • A browser hook (useLiveTranscription). Add it later if people ask.
  • Streaming text-to-speech audio. Telephony needs it too (μ-law out, sent as it's made), but it's a separate issue. generateSpeech() returns whole clips today.
  • Carrier helpers (Twilio, Telnyx and similar message parsing). Document the pattern first.
  • A full server voice loop (turn-taking, barge-in, interruption). This issue is the building block for that.

Open questions

  1. An async iterator of events, or on(event, handler) callbacks like RealtimeConnection?
  2. Should events follow AG-UI (CUSTOM events), or stay a plain transcription event union?
  3. Where does format conversion live: in @tanstack/ai, or in a small @tanstack/ai-utils audio helper?
  4. E2E: aimock has no live-audio WebSocket mocking. Use unit tests with a fake adapter until it does?

Done when

  • Activity and adapter contract in @tanstack/ai, plus two or more providers from different vendors.
  • A docs page under docs/media/, with a telephony example (a Twilio Media Streams call into chat()).
  • A media-generation skill update.
  • Changesets.
Langage dominant
TypeScript
Étoiles
3.3k
Forks
378
Merge moyen
1 j 1 h
PR mergées (30 j)
214

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de TanStack/ai

Toutes les issues de TanStack/ai

Issues similaires

Plus d'issues TypeScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.