realtime WS: commit never yields .completed for utterances >= ~1s (0.5s works)
Maintainers usually reply within 5 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- cpp
- Domain
- audio-video-rtc, backend
Research direction
Start with server/http/http_server.cpp, especially the realtime commit handler and the streaming finish path. Reproduce the issue with the raw WebSocket client using utterances from 0.5 to 2.0 seconds, then inspect how longer streams and endpointing handle finalization. Done means longer utterances emit committed and completed events, with endpointing no longer stalling.
Written by the indexing model from the issue text.
Description
Summary
The realtime WebSocket transcription path in nemo-speech serve 0.1.0 never emits .completed (or input_audio_buffer.committed) after input_audio_buffer.commit once the streamed utterance reaches ~1 second or more. At 0.5 s it works (+0.19 s finalization); at 1.0 s, 1.5 s, and 2.0 s it hangs forever. Short-utterance success shows the protocol handling is correct, so this looks length-dependent in the streaming finish path.
A second, possibly related symptom: with --asr.endpointing.enable=true and 2 s of trailing zero-PCM silence, no endpoint .completed ever fires and audio_processed stalls partway through the stream.
Environment
- Binary:
nemo-speech 0.1.0release archivenemo-speech-0.1.0-linux-x86_64-cuda.tar.gz - OS/GPU: NixOS 26.11, NVIDIA T1000 4 GB, driver 595.91, CUDA backend (
backend=CUDA0) - Model:
nemotron-speech-streaming-en-0.6b.q8_0.gguf(665 MiB) - Serve:
nemo-speech serve --asr-model <gguf> --asr.batching.state_arena_slots 1 --host 127.0.0.1 --port 8080 - Note: default arena slots OOM (
cudaMalloc failed: out of memory) with ~2 GB already resident on the 4 GB card;state_arena_slots 1was required to run at all.
Repro
Raw WebSocket client (stdlib only, masked client frames), repeated per trial on a fresh connection:
GET /v1/realtimehandshake, receivesession.created.- Send
{"type": "session.update", "session": {"sample_rate": 16000, "language": "en-US"}}, receivesession.updated. - Stream 16 kHz mono PCM16 as binary frames in 100 ms (3200-byte) chunks, back-to-back.
- Send
{"type": "input_audio_buffer.commit"}. - Read events.
Audio: 16 kHz mono command utterances; trials used 0.5 s / 1.0 s / 1.5 s / 2.0 s prefixes of the same clip.
Observed
| Streamed audio | Result after commit |
|---|---|
| 0.5 s | .completed in +0.19 s, transcript "What?" (correct for the prefix) |
| 1.0 s | no committed, no .completed, no error (25 s wait) |
| 1.5 s | same hang |
| 2.0 s | same hang |
Mid-stream .delta events flow correctly in all cases (full transcript visible within ~0.3 s of arrival, ~6x realtime), so ingestion and decoding work; only finalization never surfaces. Endpointing variant (--asr.endpointing.enable=true --asr.endpointing.stop_history_eou_ms=800, utterance + 2 s trailing silence): audio_processed advances to ~2.0-2.2 s of a 2.66 s stream then stalls; no endpoint final ever arrives.
Additional observation: streaming tail words are unstable across runs on identical input ("Play some Kendrick Lam" vs "Play some Kendrick"), and trailing silence does not flush the remainder.
Notes
- Current
main'sserver/http/http_server.cppcommit handler (ensure_stream(); emit(stream->finish()); ...committed) is identical to 0.1.0, so this likely reproduces onmain(not built here; no toolchain on the test host). - Possibly related to #48 (streaming RNNT stall) and #40/#41 (EOU handling), but this repro uses endpointing off, no silence mid-stream, and fails deterministically on utterance length alone.
- Dominant language
- C++
- Stars
- 150
- Forks
- 32
- Avg merge
- 8d 4h
- Merged PRs (30d)
- 9
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/NeMo-Speech.cpp
-
Does this runtime support multitalker-parakeet-streaming-0.6b-v1?Possibly taken @pskrunner14 claimed this 2 days ago. Openenhancement
NVIDIA/NeMo-Speech.cpp#56 · 3 comments ·
Maintainers usually reply within 5 days
-
Expose confidence estimation / token probabilities (currently always 1.0)Possibly taken @pskrunner14 claimed this 3 days ago. Openenhancement
Difficulty 4/5 3-5 days Newbie friendliness 48/100
NVIDIA/NeMo-Speech.cpp#55 · 2 comments · 1 assignee ·
Maintainers usually reply within 5 days
-
Streaming RNNT wedges after sustained zero-PCM silence; Vulkan aborts with GGML_ASSERT(ne3 == ne13)Open
Difficulty 4/5 3-5 days Newbie friendliness 48/100
NVIDIA/NeMo-Speech.cpp#48 ·
Maintainers usually reply within 5 days
-
Difficulty 5/5 Over a week Newbie friendliness 45/100
NVIDIA/NeMo-Speech.cpp#40 ·
Maintainers usually reply within 5 days
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
NVIDIA/NeMo-Speech.cpp#34 ·
Maintainers usually reply within 5 days
All issues in NVIDIA/NeMo-Speech.cpp
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
plengauer/DXGIOutputDuplication#81 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
sudoevolve/EUI-NEO#80 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Ryan-Millard/Img2Num#681 · 2 comments ·
Maintainers usually reply within 2 days
-
upstream update
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
conan-io/conan-center-index#31098 ·
Maintainers usually reply within 2 days