Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Streaming ASR leaks previous-turn punctuation into the next final (leading ? / ,)

Open
  • Is something wrong?
#22 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, grpc
Domain
audio-video-rtc, backend

Research direction

Start with the riva_server streaming endpointing/finalization and punctuation paths, using the persistent Riva gRPC stream and the listed endpointing flags to reproduce the delayed punctuation. Done means trailing punctuation is included in the preceding utterance’s final, with no punctuation-only final or leading punctuation on the next hypothesis.

Written by the indexing model from the issue text.

Description

Summary

With streaming ASR endpointing enabled on Nemotron Speech Streaming 0.6B, trailing punctuation from one utterance is not always attached to that utterance's is_final result. It arrives later as its own final, or as a leading ? / , / . on the next utterance.

Example from a live gRPC session (riva_server in nvcr.io/nvidia/nemo-speech.cpp:0.1.0):

Transcription: [Hi, can you tell me what is the capital of UK]
(user turn ends)
Transcription: [ ? What is the capital]
Transcription: [ of United States of America]

The first utterance is missing the question mark. The next utterance starts with that leftover ?.

This is not a client transcript-sort issue. The pipeline concatenates is_final chunks in receive order. The punctuation is missing from the earlier final and present on the later one.

Repro

  1. Serve ASR with endpointing on Nemotron Speech Streaming 0.6B (--asr.endpointing.enable=true, --asr.endpointing.stop_history_eou_ms=400).
  2. Keep one streaming Recognize stream open across multiple spoken turns (Riva gRPC, as a voice-agent client does).
  3. Ask a question that should end with ?, wait for the bot, then ask another question.
  4. Repeat a few turns.

Also seen with a leading comma:

Transcription: [ , okay]
Transcription: [ ? And what is the capital]

Expected

Punctuation that belongs to utterance N is included in that utterance's final, or flushed before the next speech segment starts. The next final must not start with leftover ?, ,, or . from the previous turn.

Actual

The word text finalizes first without trailing punctuation. After the next user turn has already started, a later final carries that punctuation. Sometimes it is punctuation-only ([ ?]). Sometimes it is prepended to the next hypothesis ([ ? What is the capital]). A punctuation-only final can also start a new user turn on the client (TranscriptionUserTurnStartStrategy).

Observed timing (one session, UTC):

  • 07:10:56.827 [ London] (no ?)

  • 07:10:56.829 user stopped

  • 07:11:00.482 next user started

  • 07:11:05.343 [ ?]

  • 07:12:19.572 [Hi, can you tell me what is the capital of UK] (no ?)

  • 07:12:19.573 user stopped

  • 07:12:23.288 next user started

  • 07:12:24.853 [ ? What is the capital]

Root cause

Streaming endpointing is supposed to emit one final per utterance and reset decoder state so delayed tokens do not leak. Trailing punctuation is still emitted after that reset, on the same Recognize stream.

Likely the punctuation path (self-punctuation on Nemotron RNNT, or a delayed rewrite of . / , / ?) is not flushed with the EOU final. It is applied when more audio arrives, so it lands on the next segment.

The client cannot tell the leftover ? apart from real next-turn text, because both arrive as is_final on the same stream.

Environment

  • nvcr.io/nvidia/nemo-speech.cpp:0.1.0
  • Nemotron Speech Streaming EN 0.6B GGUF (nemotron-speech-streaming-en-0.6b.q8_0.gguf)
  • Endpointing on, stop_history_eou_ms=400
  • Persistent Riva gRPC streaming Recognize (Pipecat NvidiaSTTService)
  • No separate --pnc-model passed. Punctuation is coming from the ASR path itself.

Request

Flush trailing punctuation with the utterance that produced it. Do not emit a later punctuation-only final, and do not prepend previous-turn ? / , / . onto the next hypothesis. Tracking this here so GGML riva_server and the streaming ASR client stay aligned.

Dominant language
C++
Stars
150
Forks
32
Avg merge
5d 12h
Merged PRs (30d)
11

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/NeMo-Speech.cpp

All issues in NVIDIA/NeMo-Speech.cpp

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.