Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

feat: live speech-to-text sessions for voice and telephony agents

オープン
  • 何か問題がありますか？
#1,944 コメント 0 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@AlemTuzlak がすでに取り組んでいます。

2026年10月29日 から。

評価

この issue はまだ評価されていません。

説明

waiting-on: maintainer

Problem

TanStack AI can't turn live audio into text on the server. That blocks the whole voice and telephony agent space: phone agents, call centres, IVR replacement, meeting bots, and voice agents behind an SFU.

  • generateTranscription() only takes a finished file.
  • Realtime voice (useRealtimeChat) connects the browser straight to the provider. The connection reads the microphone itself (startAudioCapture), so a server can't feed it audio from a phone call.

A server-owned voice agent needs to feed audio as it arrives, and get text back while the caller talks. The audio comes from Twilio Media Streams, Telnyx, Plivo, Vonage, LiveKit, an SFU, or a browser WebSocket. Today teams stitch this together from vendor SDKs, then hand the text to chat().

This came up from a user who runs a voice agent on Cloudflare agents with withVoice. Their reply text already comes from chat(). Speech-to-text is the piece TanStack can't provide. See Transcriber in agents/voice for one pipeline's view of the shape.

Proposal

A new activity, a live transcription session, with one provider-neutral adapter contract.

import { transcriptionSession } from '@tanstack/ai'
import { deepgramLiveTranscription } from '@tanstack/ai-deepgram'

const session = await transcriptionSession({
  adapter: deepgramLiveTranscription({ model: 'nova-3' }),
  audio: { encoding: 'mulaw', sampleRate: 8000, channels: 1 }, // a Twilio call
  language: 'en',
})

twilioSocket.on('media', (m) => session.send(m.payload)) // raw audio chunks

for await (const event of session) {
  // { type: 'interim', text }  text that may still change
  // { type: 'final', text }    stable text for a segment
  // { type: 'turn-end' }       the provider thinks the caller finished talking
  // { type: 'error', error }
}

await session.close()

Swap the adapter to change provider. Nothing else changes.

Goals
  • Runs on the server: Node, Bun, Workers and Durable Objects. No browser APIs.
  • One small adapter contract (open, send audio, receive events, close), so providers are cheap to add, including community adapters.
  • Telephony audio is first-class: G.711 μ-law and A-law at 8 kHz, plus linear PCM at 8, 16, 24 and 48 kHz. Each adapter declares what it accepts. A shared helper converts formats when the provider can't take the input as is.
  • Optional turn-end signal, when the provider has endpointing.
  • Barge-in support: the caller's speech during agent playback must reach the app fast, as interim events.
  • Small enough to wrap for other voice pipelines, for example an agents/voice Transcriber, in a few lines.
Providers

The first PR ships at least two providers from different vendors. That proves the contract isn't shaped around one API.

Provider Package Notes
Deepgram (Nova-3, Flux) new @tanstack/ai-deepgram Most common STT in telephony stacks. Flux has turn detection
AssemblyAI (Universal-Streaming) new @tanstack/ai-assemblyai Turn detection built in
ElevenLabs Scribe realtime @tanstack/ai-elevenlabs Package exists
OpenAI realtime transcription @tanstack/ai-openai Package exists
Workers AI (Deepgram models over the AI binding) @tanstack/ai-cloudflare Package exists. The user who asked
Gladia, Speechmatics, Soniox, Google, Azure community or later

I haven't confirmed live-streaming support or exact model ids for each row. Check before you build.

Out of scope for the first PR

  • A browser hook (useLiveTranscription). Add it later if people ask.
  • Streaming text-to-speech audio. Telephony needs it too (μ-law out, sent as it's made), but it's a separate issue. generateSpeech() returns whole clips today.
  • Carrier helpers (Twilio, Telnyx and similar message parsing). Document the pattern first.
  • A full server voice loop (turn-taking, barge-in, interruption). This issue is the building block for that.

Open questions

  1. An async iterator of events, or on(event, handler) callbacks like RealtimeConnection?
  2. Should events follow AG-UI (CUSTOM events), or stay a plain transcription event union?
  3. Where does format conversion live: in @tanstack/ai, or in a small @tanstack/ai-utils audio helper?
  4. E2E: aimock has no live-audio WebSocket mocking. Use unit tests with a fake adapter until it does?

Done when

  • Activity and adapter contract in @tanstack/ai, plus two or more providers from different vendors.
  • A docs page under docs/media/, with a telephony example (a Twilio Media Streams call into chat()).
  • A media-generation skill update.
  • Changesets.
主要言語
TypeScript
スター
3.3k
フォーク
378
平均マージ
1日 5時間
マージ済み PR（30日）
233

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

TanStack/ai のほかの issue

TanStack/ai の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。