Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

feat: live speech-to-text sessions for voice and telephony agents

Abierto
  • ¿Algo no está bien?
#1,944 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@AlemTuzlak ya está trabajando en esto.

Desde el 29/10/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

waiting-on: maintainer

Problem

TanStack AI can't turn live audio into text on the server. That blocks the whole voice and telephony agent space: phone agents, call centres, IVR replacement, meeting bots, and voice agents behind an SFU.

  • generateTranscription() only takes a finished file.
  • Realtime voice (useRealtimeChat) connects the browser straight to the provider. The connection reads the microphone itself (startAudioCapture), so a server can't feed it audio from a phone call.

A server-owned voice agent needs to feed audio as it arrives, and get text back while the caller talks. The audio comes from Twilio Media Streams, Telnyx, Plivo, Vonage, LiveKit, an SFU, or a browser WebSocket. Today teams stitch this together from vendor SDKs, then hand the text to chat().

This came up from a user who runs a voice agent on Cloudflare agents with withVoice. Their reply text already comes from chat(). Speech-to-text is the piece TanStack can't provide. See Transcriber in agents/voice for one pipeline's view of the shape.

Proposal

A new activity, a live transcription session, with one provider-neutral adapter contract.

import { transcriptionSession } from '@tanstack/ai'
import { deepgramLiveTranscription } from '@tanstack/ai-deepgram'

const session = await transcriptionSession({
  adapter: deepgramLiveTranscription({ model: 'nova-3' }),
  audio: { encoding: 'mulaw', sampleRate: 8000, channels: 1 }, // a Twilio call
  language: 'en',
})

twilioSocket.on('media', (m) => session.send(m.payload)) // raw audio chunks

for await (const event of session) {
  // { type: 'interim', text }  text that may still change
  // { type: 'final', text }    stable text for a segment
  // { type: 'turn-end' }       the provider thinks the caller finished talking
  // { type: 'error', error }
}

await session.close()

Swap the adapter to change provider. Nothing else changes.

Goals
  • Runs on the server: Node, Bun, Workers and Durable Objects. No browser APIs.
  • One small adapter contract (open, send audio, receive events, close), so providers are cheap to add, including community adapters.
  • Telephony audio is first-class: G.711 μ-law and A-law at 8 kHz, plus linear PCM at 8, 16, 24 and 48 kHz. Each adapter declares what it accepts. A shared helper converts formats when the provider can't take the input as is.
  • Optional turn-end signal, when the provider has endpointing.
  • Barge-in support: the caller's speech during agent playback must reach the app fast, as interim events.
  • Small enough to wrap for other voice pipelines, for example an agents/voice Transcriber, in a few lines.
Providers

The first PR ships at least two providers from different vendors. That proves the contract isn't shaped around one API.

Provider Package Notes
Deepgram (Nova-3, Flux) new @tanstack/ai-deepgram Most common STT in telephony stacks. Flux has turn detection
AssemblyAI (Universal-Streaming) new @tanstack/ai-assemblyai Turn detection built in
ElevenLabs Scribe realtime @tanstack/ai-elevenlabs Package exists
OpenAI realtime transcription @tanstack/ai-openai Package exists
Workers AI (Deepgram models over the AI binding) @tanstack/ai-cloudflare Package exists. The user who asked
Gladia, Speechmatics, Soniox, Google, Azure community or later

I haven't confirmed live-streaming support or exact model ids for each row. Check before you build.

Out of scope for the first PR

  • A browser hook (useLiveTranscription). Add it later if people ask.
  • Streaming text-to-speech audio. Telephony needs it too (μ-law out, sent as it's made), but it's a separate issue. generateSpeech() returns whole clips today.
  • Carrier helpers (Twilio, Telnyx and similar message parsing). Document the pattern first.
  • A full server voice loop (turn-taking, barge-in, interruption). This issue is the building block for that.

Open questions

  1. An async iterator of events, or on(event, handler) callbacks like RealtimeConnection?
  2. Should events follow AG-UI (CUSTOM events), or stay a plain transcription event union?
  3. Where does format conversion live: in @tanstack/ai, or in a small @tanstack/ai-utils audio helper?
  4. E2E: aimock has no live-audio WebSocket mocking. Use unit tests with a fake adapter until it does?

Done when

  • Activity and adapter contract in @tanstack/ai, plus two or more providers from different vendors.
  • A docs page under docs/media/, with a telephony example (a Twilio Media Streams call into chat()).
  • A media-generation skill update.
  • Changesets.
Lenguaje dominante
TypeScript
Estrellas
3.3k
Forks
378
Merge medio
1 d 5 h
PR fusionados (30 d)
233

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de TanStack/ai

Todos los issues de TanStack/ai

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.