Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Plugin] livekit-plugins-sherpa: local STT on sherpa-onnx (streaming + batch)

Aperta
  • Qualcosa non va?
#7,746 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@mahimairaja ci sta già lavorando.

Dal 11/10/2026.

  • #7748 di @mahimairaja — aperta

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
python
Ambito
audio-video-rtc, backend

Direzione di ricerca

An implementation is linked at livekit-plugins/livekit-plugins-sherpa in the author's repository; start by reviewing it alongside the existing silero and turn-detector plugins. The issue reports 70 tests and strict mypy checks; compare the implementation with the requested streaming and batch behavior, model downloads, and known audio limits. Done means a reviewed livekit-plugins-sherpa package that meets those requirements; an open linked pull request (#7748) is already in progress.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Feature Type

Would make my life easier

Feature Description

Every STT plugin in livekit/agents calls a hosted API. Running an agent fully on-device, in an air-gapped deployment, or just without a per-minute bill, means leaving the plugin system: an OpenAI-compatible local server behind openai.STT(base_url=...), or a one-off plugin per model.

Proposal: a livekit-plugins-sherpa package built on sherpa-onnx. One wheel, ONNX Runtime only (no torch, no GPU), and it runs most open speech models: zipformer, paraformer, Whisper, Moonshine, SenseVoice, Parakeet and more.

  • sherpa.STT: true streaming on sherpa's online recognizer. streaming=True, interim_results=True, aligned_transcript="word". The final transcript is always emitted before END_OF_SPEECH for the same utterance (the ordering #7699 depends on). Endpointing from sherpa's own rules; flush() closes an utterance and keeps word timestamps on the session clock.
  • sherpa.BatchSTT: sherpa's offline recognizer (Whisper, Moonshine, SenseVoice, Parakeet...). streaming=False; AgentSession already runs it live through stt.StreamAdapter and the default Silero VAD.
  • Built-in models are pinned to Hugging Face commits and fetched by download-files, the same way silero and turn-detector do. Any other model plugs in through recognizer=.

Tested on real models: streaming zipformer en (70M int8) and en-20M; Moonshine tiny en (int8) and Whisper tiny.en (int8). Checks run LibriSpeech clips as 48 kHz room audio, live audio through StreamAdapter + Silero VAD, and TTS speech transcribed back.

In a full local agent on an M-series Mac mini, CPU only: about 3.4 s voice to voice, with end of turn at 0.97 s.

Workarounds / Alternatives
  • openai.STT(base_url=...) against a local server: an extra service to run, and no streaming interims or word timings.
  • Per-model plugins such as FunASR/SenseVoice in #5897 (PR #6176), batch only. sherpa-onnx also runs SenseVoice, so one plugin could cover that family plus streaming models.
  • #6522 (PR #6525) composes a local example from existing plugins; this would give it a native STT.
Additional Context

Working implementation, 70 tests, mypy strict: https://github.com/mahimairaja/local-livekit (livekit-plugins/livekit-plugins-sherpa)
Write-up with measurements: https://mahimai.ca/blog/local-livekit-voice-agent-offline

Known limits: the built-in wideband models return empty text for clean 8 kHz (telephony) audio, so SIP needs an 8 kHz model via recognizer=; the 20M streaming model drops leading words on resampled 48 kHz audio, so it is not the default.

TTS side of the same package: #7747.

Happy to open a PR adding the package under livekit-plugins/.

Lingua principale
Python
Stelle
14.6k
Fork
3.9k
Merge medio
3g 1h
PR unite (30g)
166

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di livekit/agents

Tutte le issue di livekit/agents

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.