Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

AutoModel passes the original source rate to already-resampled speaker segments

Abierto Apto para principiantes
  • ¿Algo no está bien?
#3,762 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
72/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, pytorch
Área
audio-video-rtc

Línea de trabajo

Empieza en funasr/auto/auto_model.py, en el límite del modelo de hablante de AutoModel.generate, y rastrea en qué se diferencia la tasa del segmento remuestreado de la fs original. Ejecuta el script de reproducción proporcionado para registrar el número de muestras que entran en la extracción de características del hablante. La tarea se considera terminada cuando el límite del hablante usa la tasa real del segmento, la reproducción da 24.000 muestras y pasan las pruebas de regresión específicas existentes.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

🐛 Bug

On main commit 66d7a4c264a5993a2a63ed00c1f402c296ee521a, the VAD pipeline resamples source audio to the ASR frontend rate before slicing and speaker chunking. The ASR boundary explicitly uses the segment rate, but the speaker boundary forwards the original generate(..., fs=...) value. CAMPPlus uses this as its source rate and resamples the waveform again.

A 200 ms synthetic 48 kHz input becomes a 16 kHz segment, padded by sv_chunk to 1.5 seconds (24,000 samples). Speaker feature extraction instead receives 8,000 samples. With an 8 kHz source override the same padded segment becomes 48,000 samples. An incompatible spk_kwargs.fs can also override the actual segment rate.

PR #3751 fixed the VAD→ASR rate propagation and retained the original rate across input batches; this report concerns the remaining speaker boundary.

To Reproduce

Use a source checkout at the SHA above and a local Python 3.12 environment. The following script uses actual AutoModel generation, audio resampling, speaker chunking and CAMPPlus inference, with small inference stand-ins and a feature-capture boundary. It downloads no model weights.

Tested install:

python -m pip install torch==2.10.0 torchaudio==2.10.0 numpy==1.26.4 transformers==4.51.3 einops pytorch-wpe -e .

Save the code below as reproduce_speaker_rate.py, then run:

HF_HUB_OFFLINE=1 MODELSCOPE_OFFLINE=1 OMP_NUM_THREADS=1 python reproduce_speaker_rate.py

Code sample

"""Weights-free public-entry reproduction; real loading/resampling and CAMPPlus entry."""

from types import SimpleNamespace
from unittest.mock import patch

import numpy as np
import torch

from funasr.auto.auto_model import AutoModel
from funasr.models.campplus import model as campplus_model


class Probe(torch.nn.Module):
    def __init__(self, role):
        super().__init__()
        self.anchor = torch.nn.Parameter(torch.zeros(1))
        self.role = role

    def inference(self, data_in, key, **kwargs):
        if self.role == 'speaker':
            print('speaker received fs:', kwargs.get('fs', 16000))
            return campplus_model.CAMPPlus.inference(self, data_in, key=key, **kwargs)
        results = [
            {'key': k, 'value': [[0, 200]]} if self.role == 'vad'
            else {'key': k, 'text': 'probe'} for k in key
        ]
        return results, {'batch_data_time': 0.2}

    def forward(self, features):
        return torch.ones(features.shape[0], 4)


captured = []


def capture_features(audio):
    captured.extend(len(sample) for sample in audio)
    return torch.zeros(len(audio), 20, 80), [20] * len(audio), captured[-len(audio):]


wrapper = AutoModel.__new__(AutoModel)
wrapper.model = Probe('asr')
wrapper.vad_model = Probe('vad')
wrapper.spk_model = Probe('speaker')
wrapper.punc_model = None
wrapper.kwargs = {
    'device': 'cpu', 'disable_pbar': True, 'return_spk_res': False,
    'frontend': SimpleNamespace(fs=16000), 'ncpu': 1,
}
wrapper.vad_kwargs = {'device': 'cpu'}
wrapper.spk_kwargs = {'device': 'cpu'}
wrapper._store_base_configs()
source_rate = 48000
audio = np.sin(2 * np.pi * 440 * np.arange(source_rate // 5) / source_rate).astype(np.float32)
with patch.object(campplus_model, 'extract_feature', capture_features):
    wrapper.generate(input=audio, fs=source_rate)
print('sample counts entering speaker feature extraction:', captured)
print('expected: [24000] (1.5 seconds padded at 16 kHz)')
assert captured == [24000], captured

Expected behavior

The speaker model receives the actual rate of its input segments (fs=16000 in this pipeline) and feature extraction sees 24,000 samples. An explicit source rate still reaches VAD/source loading and constructor configuration remains available for the next call.

Error logs

speaker received fs: 48000
sample counts entering speaker feature extraction: [8000]
expected: [24000] (1.5 seconds padded at 16 kHz)
AssertionError: [8000]

The proposed one-line boundary fix produces fs: 16000 and [24000] on the same script. New focused regressions fail 21 cases on the baseline while 8 controls pass; the fixed source passes all 29. Tests cover arrays/explicit PCM, single/batch inputs, constructor/runtime precedence, speaker config conflicts and subsequent calls. No recognition or diarization quality claim is made from these stand-in models.

Environment

  • OS: macOS 26.6.2 arm64
  • Python: 3.12.12
  • FunASR: source 1.4.16, SHA above
  • ModelScope: 1.40.1
  • PyTorch / torchaudio: 2.10.0 / 2.10.0
  • NumPy: 1.26.4; transformers: 4.51.3; tokenizers: 0.21.4
  • Install method: editable source in a task-local virtual environment
  • Device: CPU; no GPU/CUDA/Docker execution

Audio details

Synthetic mono float32 440 Hz sine, 200 ms, source rate 48 kHz; additional regressions use 8/16 kHz and explicit signed PCM16. Speaker chunking pads it to 1.5 seconds. There is no speech, real speaker identity or background audio.

Prepared with AI assistance; this is a current-source reproduction, not a recovered historical patch.

Lenguaje dominante
Python
Estrellas
20.6k
Forks
2.1k
Merge medio
18 h 3 min
PR fusionados (30 d)
88

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de modelscope/FunASR

Todos los issues de modelscope/FunASR

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.