Add Nemotron Unified ASR 0.6B (25 European languages) as an opt-in streaming model for German + English dictation

Open
  • Is something wrong?
#1,682 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
34/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
machine-learning, rust
Domain
audio-video-rtc, backend

Research direction

Start by reading the existing Parakeet Unified English model integration and the linked transcribe.cpp#250 request, then check how models are exposed in the picker. The issue asks for Nemotron Unified as an opt-in streaming model with a final pass on the same model. Measure Auto behavior on German and English clips that switch languages and at openings, comparing results with Parakeet Ultra; the payload does not name test files or an existing test command.

Written by the indexing model from the issue text.

Description

What are you trying to do?

I dictate all day in German and English, often both in one dictation, on macOS (M4 Pro). Today I choose between live text and accuracy. Nemotron 3.5 streams, but it lost or garbled the opening of 6 of 33 of my dictations on Auto (#1517). Parakeet Ultra on Auto got all 33 right and is what I use now, but it doesn't stream.

NVIDIA released nvidia/nemotron-labs-unified-asr-0.6b on 27.10. It's the multilingual version of the Parakeet Unified English that Hex already ships: the same 0.6B unified FastConformer-RNNT, the same buffered streaming with 5.6 s left context, chunk and right context set from 2.08 s down to 0.16 s, punctuation and capitalization, and 25 European languages behind a language-ID prompt (target_lang="de", "en", … or "auto"). License OpenMDW-1.1, like Nemotron 3.5.

From its card, WER % with the language given:

                                 offline  1.12 s  0.56 s  0.32 s  0.16 s
German (FLEURS+MCV+MLS average)   4.32     5.45    6.30    7.47   10.01
English, 7 cleaned sets           4.68     5.41    5.68    6.27    7.87
Nemotron 3.5 streaming, English    -       7.26    7.58    7.97    8.90
Parakeet v3, English (offline)    4.93

FLEURS German alone is 3.87 offline and 5.24 at 1.12 s. Nemotron 3.5 streaming's card gives about 8.2 at 1.12 s. So at 1.12 s it streams German at about v3's offline level, and well ahead of Nemotron 3.5.

Mac ports already exist and look fast enough:

It isn't in transcribe.cpp yet. That's requested in handy-computer/transcribe.cpp#250.

What would help?

Nemotron Unified as an opt-in model in the picker, next to Parakeet Unified English, once transcribe.cpp carries it. Streaming for dictation, with a final pass on the same model.

One thing needs measuring before it's recommended for mixed-language dictation. The card's numbers are all with an explicit language. It warns that on auto at latencies of 0.56 s or less, "the first chunks do not contain enough speech" and it "can begin emitting tokens belonging to a different language", which can't be revised. That's the #1517 failure. It says the effect is "much smaller" at higher latency, but publishes no auto numbers. So:

  • Default it to auto at 1.12 s (or 2.08 s) for users on Auto, not a low-latency setting.
  • Check auto on German + English clips that switch mid-sentence, and on openings, against Parakeet Ultra's final pass. The #1520 fallback (re-transcribe when the live text loses its start or switches language) would catch the rest.

I can run it on my saved dictations (Keep recordings) as soon as there's a dev build, and report openings lost and German error against Ultra, as I did on #1517.

Dominant language
Rust
Stars
5.7k
Forks
337
Avg merge
2h 57m
Merged PRs (30d)
891

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from anomalyco/hex

All issues in anomalyco/hex

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.