[Bug] SenseVoice Chinese text clipping misses multi-character queries and uses incorrect timestamp indices

funclip/utils/trans_utils.py、特に _normalize_recognition_result、pre_proc、proc から始め、次に VideoClipper が認識状態をクリッピングに渡す方法を追跡します。モデルをダウンロードせずに issue の簡潔な再現を実行します。複数文字および 1 文字の中国語クエリが指定されたタイムスタンプ範囲に対応し、句読点、混在言語のテキスト、繰り返しの一致、既存の ASCII の大文字・小文字の動作が維持されれば完了です。

Summary

SenseVoice can produce Chinese transcription, token timestamps and SRT successfully, yet selecting that same Chinese text for audio clipping fails. The saved raw transcript is contiguous Chinese, while the query is converted to space-separated characters; single-character matches also use the wrong timestamp index.

Environment

• FunClip: current main 9e720617949e32da4c0a564d10cf5a64337fe2c2, with relevant source files verified unchanged.
• FunASR: 1.4.15.
• Linux x86-64; Python 3.12.3; isolated venv with the repository requirements.
• torch/torchaudio 2.10.0+cpu, NumPy 1.26.4, librosa 0.11.0, soundfile 0.14.0, ModelScope 1.40.0.
• Gradio 4.44.1 installed, but this reproduction calls the Python API, not a browser.
• Actual CPU model loading and inference; no ASR stand-in in the original acceptance run.

Audio or video input

The original acceptance used an 18-second, 16 kHz mono Chinese WAV with actual create_asr_model("sensevoice", "zh", ...), SenseVoiceSmall, FSMN VAD and CAM++. No hotwords or LLM. All 40 cached model files matched public ModelScope metadata size/SHA-256.

The small transcript/timestamp excerpt below is sufficient to reproduce the matching defect without downloading models or sharing the audio. It is an excerpt of saved real recognition state, not an acoustic accuracy reference.

Steps to reproduce

From the repository root in an installed environment:

import sys
sys.path.insert(0, "funclip")
from utils.trans_utils import pre_proc, proc

raw = "嗯那么今天我们就简"
timestamps = [
    [230, 290], [590, 650], [710, 770],
    [890, 950], [1010, 1070], [1250, 1310],
    [1430, 1490], [1610, 1670], [1970, 2030],
]
for query in (raw[:8], "简"):
    print(repr(query), repr(pre_proc(query)),
          proc(raw, timestamps, pre_proc(query)))

# Diagnostic control only: not a production fix.
for query in (raw[:8], "简"):
    print("token-spaced control",
          proc(pre_proc(raw), timestamps, pre_proc(query)))

In the original real-model run, VideoClipper.recog((16000, data), sd_switch="No") returned 76 characters, 76 timestamps and three SRT entries. Calling clipper.clip("嗯那么今天我们就", 0, 0, state) with that untouched state returned the full input instead of a selected excerpt.

Expected behavior

Queries copied from the recognized text should map to their corresponding timestamps regardless of whether the model's raw Chinese text includes token-separating spaces.

For the excerpt above, the first eight characters map to 230-1670 ms, or sample indices [3680, 26720] at 16 kHz. The ninth character maps to 1970-2030 ms, or [31520, 32480]. These are expected mappings to the model-returned timestamps, not a claim that the timestamps are acoustically perfect.

Actual behavior

The multi-character query returns no ranges. The single-character query finds the character but maps it to the first timestamp, 230-290 ms.

The actual full audio-API run returned:

No period found in the speech, return raw speech. You may check the recognition result and try other destination text.

The serialized PCM16 WAV was still 288,000 samples / 18 seconds at 16,000 Hz, and the clip SRT was empty. Recognition/SRT generation succeeded; end-to-end text clipping did not.

Logs or traceback

Minimal matching output:

'嗯那么今天我们就' '嗯 那 么 今 天 我 们 就' []
'简' '简' [[3680, 4640]]
token-spaced control [[3680, 26720]]
token-spaced control [[31520, 32480]]

There is no matching exception. The real acceptance harness exited with an AssertionError because the clip message reported no match and the output was the original audio.

The mismatch is in the contract between _normalize_recognition_result, pre_proc and proc: normalized raw text can fall back to continuous text/text_tn; query preprocessing inserts spaces; matching searches the un-tokenized raw string, and its timestamp index counts raw spaces. Inserting spaces into the saved raw text explains this particular failure, but is only a diagnostic control. A repair must preserve token/timestamp alignment, punctuation, mixed-language text, repeated matches and the existing ASCII case behavior.

Screenshots or clips

The returned transcript excerpt, model timestamps, matcher outputs and serialized-WAV measurements above demonstrate the issue. No browser/video-export or speaker-diarization acceptance is claimed; the speaker-enabled case was not reached after the plain clipping assertion failed.

Related issues were checked before filing: #161 resolved rich-tag/subtitle-return handling; #88 requested SenseVoice support; #198 fixed English case matching and has reporter confirmation; #60 discussed long multi-selection input without this concrete model-output reproduction. #214 is a separate non-16 kHz normalization defect; this run uses 16 kHz throughout. Those issues are not being reopened.

This report was prepared with Codex assistance. No production repair or release is claimed.