[Feature Request] Allow VAD to run on a different device than the ASR model (Apple Silicon MPS regression: VAD 5x slower than CPU)

Inizia in funasr/auto/auto_model.py intorno alle righe 470, 483 e 499, quindi traccia inference_with_vad() e il modo in cui i tensori delle feature raggiungono il modello VAD. Confronta il rispetto di un vad_kwargs["device"] esplicito con il design vad_device proposto, mantenendo ASR sul dispositivo principale. Il lavoro è completato quando VAD e ASR funzionano su dispositivi misti senza il fragile workaround di monkey-patching; verificalo con la riproduzione CPU/MPS fornita e il confronto dei tempi.

Summary

AutoModel forces vad_kwargs["device"] to equal the main ASR model's device (funasr/auto/auto_model.py:470). There is no way to run VAD on CPU while the ASR model runs on GPU/MPS. On Apple Silicon this is a measurable ~5x performance regression for the VAD stage, because the FSMN streaming VAD emits many tiny per-frame forwards that suffer from MPS per-op kernel-launch overhead.

Environment

• macOS / Apple M2 (8-core, 16GB)
• funasr 1.4.14
• torch 2.14.0, torch.backends.mps.is_available() = True

Measured impact

Same 121-minute (7285s) audio, FSMN VAD only:

Device	VAD wall-clock	Realtime factor
cpu	23.6s	308x
mps	123.0s	59x

VAD on MPS is 5.2x slower than CPU. Full pipeline (paraformer-large + fsmn-vad + ct-punc), same 121-min audio:

Config	Total	Notes
device='mps' (VAD+ASR both on MPS)	~259s	VAD=123s, ASR=91s
VAD on CPU + ASR on MPS (mixed)	~156s	VAD=24s, ASR=91s

Mixed device saves ~40% end-to-end. ASR (paraformer) genuinely benefits from MPS (large batched matmuls); VAD does not.

Root cause

funasr/auto/auto_model.py (1.4.14):

# AutoModel.__init__, ~line 465-470
vad_kwargs = {} if kwargs.get("vad_kwargs", {}) is None else kwargs.get("vad_kwargs", {})
if vad_model is not None:
    vad_kwargs["model"] = vad_model
    vad_kwargs["model_revision"] = kwargs.get("vad_model_revision", "master")
    vad_kwargs["device"] = kwargs["device"]   # <-- hardcoded to main device

So even passing vad_kwargs={"device": "cpu"} is overwritten. inference_with_vad() then runs self.inference(model=self.vad_model, kwargs=self.vad_kwargs) with vad_kwargs["device"] fixed to the main device, so feature tensors land on the main device and VAD weights must match.

Feature request

Expose a way to place the VAD model on a device independent of the ASR model, e.g.:

AutoModel(model=..., vad_model=..., punc_model=..., device='mps', vad_device='cpu')

which would build the VAD on vad_device and move feature tensors fed to self.vad_model onto vad_device before VAD forward, while the ASR model stays on device. (punc/spk sub-models have the same hardcoded coupling at lines 483/499, so a general per-submodel device would be ideal.)

This matters most on Apple Silicon today, but the pattern (streaming VAD = many tiny forwards) is device-agnostic: any accelerator with high per-op launch overhead is hurt by forcing VAD onto it.

Minimal reproduction

import time
from pathlib import Path
from funasr import AutoModel

models = [Path.home()/'.cache/modelscope/models'/('iic--'+n)/'snapshots/master' for n in [
    'speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    'speech_fsmn_vad_zh-cn-16k-common-pytorch',
    'punc_ct-transformer_cn-en-common-vocab471067-large',
]]
for dev in ('cpu', 'mps'):
    m = AutoModel(model=str(models[1]), device=dev, disable_update=True, disable_pbar=True)
    t = time.monotonic()
    m.generate(input='your_16k_mono.wav', max_single_segment_time=60000)
    print(dev, f'{time.monotonic()-t:.1f}s')

Workaround (1.4.14)

Build the main model on the accelerator, then replace MODEL.vad_model with a separately-built CPU VAD instance and patch ComputeScores to move feature tensors onto the CPU device:

main = AutoModel(model=asr, vad_model=vad, punc_model=punc, device='mps', ...)
cpu_vad = AutoModel(model=vad, device='cpu', ...).model
_orig = cpu_vad.ComputeScores
cpu_vad.ComputeScores = lambda feats, cache=None: _orig(feats.to('cpu') if hasattr(feats,'to') and feats.device.type!='cpu' else feats, cache=cache)
main.vad_model = cpu_vad
main.vad_kwargs['device'] = 'cpu'

Works (verified, 156s vs 259s) but fragile across versions, hence this request.

Happy to turn this into a PR — the design question is whether to honor vad_kwargs["device"] when explicitly provided (smallest change) vs. add a dedicated vad_device argument.