Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Inference: xai/stt-2 — keyterm support, late final transcripts, and missing word-alignment entry

Aperta
  • Qualcosa non va?
#7,467 1 commento 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend-api-design

Direzione di ricerca

Start with livekit/agents/inference/stt.py, where the issue says _keyterms_extra_for_model and _WORD_ALIGNED_MODELS need updates; compare the existing xai/stt-1 handling with xai/stt-2. Then check how Inference decodes xAI ExtraSettings and how the related flush-at-turn-boundary work in #7285 fits. Done means keyterms reach xAI, xai/stt-2 is recognized as word-aligned, and tests cover the supported behavior; the transcript-timing questions may need separate investigation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Feature Type

Would make my life easier

Feature Description

We'd like to run xai/stt-2 (Grok Voice Transcribe 2.0) through LiveKit Inference as the primary STT in a cascaded pipeline. Three things block it today (tested on livekit-agents 1.8.3 against the Inference gateway, 2026-09-25):

1. No keyterm/vocabulary support for xAI on Inference.
xAI's streaming API natively accepts keyterm (up to 100 terms), and the direct livekit-plugins-xai STT exposes it (#6587). Through Inference, the xAI ExtraSettings only accepts diarize, endpointing, format and interim_results. Every vocabulary key we tried fails the whole session rather than being ignored:

LiveKit Inference STT returned error: failed to create session: failed to decode extra settings:
'xai.ExtraSettings' has invalid keys: keyterm   (status_code=2006)

(Same for keyterms, keywords, custom_vocabulary, vocabulary, prompt, context, smart_turn, vad_threshold.) Framework-managed keyterms (stt_context_options / session.update_options(keyterms=...)) also skip xAI, because _keyterms_extra_for_model has no xAI mapping. For call flows that capture proper nouns (customer names, places), this is the deciding gap. On the same audio, keyterm-biased deepgram/nova-3 got "Aoife Nic Ghearailt" exactly, while xai/stt-2 without vocabulary returned "Awif Nick Gerelt".

Ask: forward keyterm for xai/* on Inference, and add the mapping in _keyterms_extra_for_model so managed keyterms reach it. Exposing smart_turn / smart_turn_timeout would help with point 2.

2. Final transcripts arrive late and irregularly.
Short conversational turns with 2.5 s pauses, real-time paced, same audio for both models:

Final transcript, measured from end of speech 1: "Yes, that's right." 2: "The reference is L C A zero zero three five." 3: "No, I don't need anything else, thanks."
deepgram/nova-3 +0.28–0.31 s +0.26–0.31 s +0.27–0.31 s
xai/stt-2 "Yes." +0.76–1.15 s, "That's right." ~+1.4 s not within the pause; arrives only after turn 3 ends (~6 s late) first sentence +1.3 s; "Thanks." only when the stream closes

(Two runs each, consistent.)

Setting endpointing (10 / 300 ms) didn't change this. With the audio turn detector committing the turn at the pause, the committed transcript is missing its tail. Related: #7285 (flush STT at detected turn boundaries). Does Inference forward a flush/finalize to xAI, and could smart_turn_timeout be exposed to bound it?

3. xai/stt-2 is missing from _WORD_ALIGNED_MODELS.
In livekit/agents/inference/stt.py (1.8.3 and main), xai/stt-1 is listed but xai/stt-2 isn't, so its aligned_transcript capability is False. In a stt.FallbackAdapter that makes the whole adapter unaligned, which silently turns adaptive interruption off for the session. xai/stt-2 does return word timings through Inference (48 timed words in our test), so it looks like it only needs adding to the set and to the XaiModels literal.

Workarounds / Alternatives
  • The direct livekit-plugins-xai STT supports keyterm and smart_turn_timeout, but it needs a separate xAI key and bill (we consolidated on Inference), and #6652 (session-managed keyterms) and #7279 (stream stops silently after a server-side error) are still open.
  • For now we keep deepgram/nova-3 + keyterms as primary.
Additional Context
  • livekit-agents 1.8.3, Python, LiveKit Cloud (EU), Inference gateway.
  • Test audio: Deepgram Aura TTS, English, fed at real-time pace in 20 ms frames.
  • Related: #6587, #6652, #7279, #7285, #7441.
Lingua principale
Python
Stelle
14.6k
Fork
3.9k
Merge medio
2g 6h
PR unite (30g)
153

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di livekit/agents

Tutte le issue di livekit/agents

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.