Nemotron multilingual streaming: the 2240ms tier is far slower than 560ms/1120ms with default compute units (about 3x to 10x real time vs 23x to 45x, about 2 GB resident)

Aperta
  • Qualcosa non va?
#1,116 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
macos, swift
Ambito
machine-learning, performance

Direzione di ricerca

Start by comparing the 2240ms and 1120ms encoder compute plans with MLComputePlan per-op device usage, then inspect StreamingNemotronMultilingualAsrManager.swift:726-739 for the existing encoder compute-unit override. Reproduce the reported tiers with the default configuration and compare them with CPU_AND_GPU; done means identifying why the 2240ms tier is slower and documenting a verified resolution or an accurate performance description in NemotronChunkSize.swift and NemotronMultilingual.md.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

FluidAudio 0.18.2, StreamingNemotronMultilingualAsrManager, default configuration. Apple M3 Pro, 36 GB, macOS 27.3 (build 26C5035r, a beta; see the note at the end).

Summary

With no configuration passed, all three tiers load with .cpuAndNeuralEngine (the manager's documented default, MLModelConfigurationUtils.defaultConfiguration()). The 560ms and 1120ms tiers load in about 13 s, stream faster than real time by 23x and 35x, and stay around 130 MB resident. The 2240ms tier takes about 200 to 240 s to load the first time (later loads 19 to 68 s), streams at about 3x to 10x, and holds about 2 GB resident. NemotronMultilingual.md says 2240ms is a shipped tier and the Nemotron chunk-size comment describes it as the highest throughput one.

Repro

FluidAudio only, defaults only. TIER is the folder of one tier of the multilingual build (.../nemotron-multilingual/multilingual/<560ms|1120ms|2240ms>), clip.wav is the same 27.5 s English clip for every run: FluidInference/fleurs en_us_0000, en_us_0003 and en_us_0001 joined with 0.7 s of silence and 1.0 s of leading silence, 48 kHz mono.

import FluidAudio
import Foundation

let samples = try AudioConverter().resampleAudioFile(URL(fileURLWithPath: "clip.wav"))   // 16 kHz
let mgr = StreamingNemotronMultilingualAsrManager()               // defaults
let t0 = Date()
try await mgr.loadModels(from: URL(fileURLWithPath: TIER))
let load = Date().timeIntervalSince(t0)
await mgr.setLanguage("en-US")
let t1 = Date()
var i = 0
while i < samples.count {                                          // 100 ms chunks
    let j = min(i + 1600, samples.count)
    _ = try await mgr.process(samples: Array(samples[i..<j]))
    i = j
}
_ = try await mgr.finish()
let dur = Date().timeIntervalSince(t1)
print("load=\(load)s process=\(dur)s RTFx=\(27.48 / dur)")

Observed (one process per run, nothing else running, resident size read with task_info after finish()):

tier=560ms  cu=cpuAndNeuralEngine load=13.2s  process=1.2s  audio=27.5s RTFx=23.3 rss=130MB
tier=1120ms cu=cpuAndNeuralEngine load=14.0s  process=0.8s  audio=27.5s RTFx=35.3 rss=126MB
tier=2240ms cu=cpuAndNeuralEngine load=202.8s process=10.3s audio=27.5s RTFx=2.7  rss=2026MB   (first run)
tier=2240ms cu=cpuAndNeuralEngine load=67.8s  process=3.0s  audio=27.5s RTFx=9.2  rss=2143MB   (second run)

With only the encoder moved off the Neural Engine through the existing FLUIDAUDIO_ENCODER_CU override (StreamingNemotronMultilingualAsrManager.swift:726-739):

FLUIDAUDIO_ENCODER_CU=CPU_AND_GPU, tier=2240ms: load=7.7s process=1.1s RTFx=24.5 rss=1797MB

A second, independent run on the same machine, warm caches in both cases: default 2240ms load=18.6s RTFx=9.9 rss=2033MB; FLUIDAUDIO_ENCODER_CU=CPU_AND_GPU 2240ms load=3.7s RTFx=27.4 rss=1770MB (the 1120ms tier gave load=13.5s RTFx=44.5). The cu= the repro prints is the manager's configuration, not the encoder override.

So the same files run at the speed of the other tiers when the encoder is not placed on the Neural Engine. The 560ms and 1120ms tiers on the same machine and OS do not show this.

What I could not determine

  • Whether the 2240ms encoder falls off the Neural Engine at load (the slow, CPU-light load suggests a long ANE compile or a fallback), or whether it runs there slowly. I did not profile placement.
  • Whether this is specific to this OS build (macOS 27 beta). I could not run another OS here. The two smaller tiers behave on it, which suggests the 2240ms graph rather than the OS.
  • I did not test the default Application Support location; the model folders were read from a separate directory with the same contents, which should not matter.

Suggested next step

Check the 2240ms encoder's compute plan (MLComputePlan per-op device usage) against the 1120ms one. If it cannot be made to stay on the ANE, consider .cpuAndGPU as the default for that tier, or stop describing it as the highest-throughput tier in NemotronChunkSize.swift:5 and NemotronMultilingual.md.

Lingua principale
Swift
Stelle
3k
Fork
451
Merge medio
2g 3h
PR unite (30g)
90

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di FluidInference/FluidAudio

Tutte le issue di FluidInference/FluidAudio

Issue simili

Altre issue su Swift

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.