[Feature Request] Allow VAD to run on a different device than the ASR model (Apple Silicon MPS regression: VAD 5x slower than CPU)

Bắt đầu trong funasr/auto/auto_model.py quanh các dòng 470, 483 và 499, sau đó theo dõi inference_with_vad() và cách các tensor đặc trưng đi tới mô hình VAD. So sánh việc tôn trọng một vad_kwargs["device"] tường minh với thiết kế vad_device được đề xuất, đồng thời giữ ASR trên thiết bị chính. Công việc được xem là hoàn tất khi VAD và ASR hoạt động trên các thiết bị khác nhau mà không cần workaround monkey-patching mong manh; hãy xác minh bằng bản tái hiện CPU/MPS được cung cấp và so sánh thời gian.

Summary

AutoModel forces vad_kwargs["device"] to equal the main ASR model's device (funasr/auto/auto_model.py:470). There is no way to run VAD on CPU while the ASR model runs on GPU/MPS. On Apple Silicon this is a measurable ~5x performance regression for the VAD stage, because the FSMN streaming VAD emits many tiny per-frame forwards that suffer from MPS per-op kernel-launch overhead.

Environment

• macOS / Apple M2 (8-core, 16GB)
• funasr 1.4.14
• torch 2.14.0, torch.backends.mps.is_available() = True

Measured impact

Same 121-minute (7285s) audio, FSMN VAD only:

Device	VAD wall-clock	Realtime factor
cpu	23.6s	308x
mps	123.0s	59x

VAD on MPS is 5.2x slower than CPU. Full pipeline (paraformer-large + fsmn-vad + ct-punc), same 121-min audio:

Config	Total	Notes
device='mps' (VAD+ASR both on MPS)	~259s	VAD=123s, ASR=91s
VAD on CPU + ASR on MPS (mixed)	~156s	VAD=24s, ASR=91s

Mixed device saves ~40% end-to-end. ASR (paraformer) genuinely benefits from MPS (large batched matmuls); VAD does not.

Root cause

funasr/auto/auto_model.py (1.4.14):

# AutoModel.__init__, ~line 465-470
vad_kwargs = {} if kwargs.get("vad_kwargs", {}) is None else kwargs.get("vad_kwargs", {})
if vad_model is not None:
    vad_kwargs["model"] = vad_model
    vad_kwargs["model_revision"] = kwargs.get("vad_model_revision", "master")
    vad_kwargs["device"] = kwargs["device"]   # <-- hardcoded to main device

So even passing vad_kwargs={"device": "cpu"} is overwritten. inference_with_vad() then runs self.inference(model=self.vad_model, kwargs=self.vad_kwargs) with vad_kwargs["device"] fixed to the main device, so feature tensors land on the main device and VAD weights must match.

Feature request

Expose a way to place the VAD model on a device independent of the ASR model, e.g.:

AutoModel(model=..., vad_model=..., punc_model=..., device='mps', vad_device='cpu')

which would build the VAD on vad_device and move feature tensors fed to self.vad_model onto vad_device before VAD forward, while the ASR model stays on device. (punc/spk sub-models have the same hardcoded coupling at lines 483/499, so a general per-submodel device would be ideal.)

This matters most on Apple Silicon today, but the pattern (streaming VAD = many tiny forwards) is device-agnostic: any accelerator with high per-op launch overhead is hurt by forcing VAD onto it.

Minimal reproduction

import time
from pathlib import Path
from funasr import AutoModel

models = [Path.home()/'.cache/modelscope/models'/('iic--'+n)/'snapshots/master' for n in [
    'speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    'speech_fsmn_vad_zh-cn-16k-common-pytorch',
    'punc_ct-transformer_cn-en-common-vocab471067-large',
]]
for dev in ('cpu', 'mps'):
    m = AutoModel(model=str(models[1]), device=dev, disable_update=True, disable_pbar=True)
    t = time.monotonic()
    m.generate(input='your_16k_mono.wav', max_single_segment_time=60000)
    print(dev, f'{time.monotonic()-t:.1f}s')

Workaround (1.4.14)

Build the main model on the accelerator, then replace MODEL.vad_model with a separately-built CPU VAD instance and patch ComputeScores to move feature tensors onto the CPU device:

main = AutoModel(model=asr, vad_model=vad, punc_model=punc, device='mps', ...)
cpu_vad = AutoModel(model=vad, device='cpu', ...).model
_orig = cpu_vad.ComputeScores
cpu_vad.ComputeScores = lambda feats, cache=None: _orig(feats.to('cpu') if hasattr(feats,'to') and feats.device.type!='cpu' else feats, cache=cache)
main.vad_model = cpu_vad
main.vad_kwargs['device'] = 'cpu'

Works (verified, 156s vs 259s) but fragile across versions, hence this request.

Happy to turn this into a PR — the design question is whether to honor vad_kwargs["device"] when explicitly provided (smallest change) vs. add a dedicated vad_device argument.