Fine-tuning Fun-ASR-Nano on a B200: one step in six takes 1 s because cuDNN SDPA builds a plan for every new batch shape

Comienza con examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md y la receta finetune.sh, y después revisa el comportamiento de cuDNN SDPA y la solución alternativa indicados. Documenta el síntoma de los pasos lentos, las GPU sm_90/sm_100 afectadas y la solución alternativa TORCH_CUDNN_SDPA_DEPRIORITIZED=1; el cambio estará terminado cuando un fine-tuner pueda identificar y evitar el problema sin hacer profiling.

Before asking

1. Searched existing issues for "cudnn", "sdpa", "slow step", "finetune Fun-ASR-Nano". No match.
2. Searched the docs and examples/industrial_data_pretraining/fun_asr_nano/docs/finetune.md.
3. Read the README quick start and docs/training.md.

Question

I ran the Fun-ASR-Nano fine-tuning recipe (examples/industrial_data_pretraining/fun_asr_nano/finetune.sh, LLM fine-tuning with the encoder and adaptor frozen, one process) on one B200, following the install steps in docs/installation/installation.md and docs/finetune.md, and got 0.245 s per step on average: 323 of 382 steps took 0.098 s and 59 took about 1.05 s. A profile showed that the slow steps are the ones whose (batch, padded token length) the process had not seen before: the Qwen3 attention calls scaled_dot_product_attention with an explicit padding mask (transformers/integrations/sdpa_attention.py:124), torch 2.11 sends that to cuDNN first on sm_100, and cuDNN builds a new execution plan per shape, 0.44 s forward and 0.55 s backward. With cuDNN taken out of the SDPA backend order (torch.backends.cuda.enable_cudnn_sdp(False) at model init, or TORCH_CUDNN_SDPA_DEPRIORITIZED=1 in the environment), the same step took 0.100 s on average, with the loss within bf16 rounding of the unmodified run.

Code or command

cd examples/industrial_data_pretraining/fun_asr_nano
export CUDA_VISIBLE_DEVICES=0
bash finetune.sh   # with a local model dir, my AISHELL-1 jsonl (3600 train / 200 dev utterances), max_epoch=1, seed=1234

What have you tried?

The trainer's per-step log prints forward_time and backward_time, and both jump by about 0.5 s on the slow steps, but nothing in the log or in docs/finetune.md connects that to the batch shape or to the attention backend, and the step-time histogram only shows up if one plots the 382 lines. A one-line note in docs/finetune.md (or in the trainer's log when a step is several times the running median) saying that on sm_90 and sm_100 cuDNN SDPA rebuilds its plan for every new batch shape, and that TORCH_CUDNN_SDPA_DEPRIORITIZED=1 avoids it, would have saved me the profile.

I opened #3705 to record my experiment: setup, measurements, traces and some potential fixes. A different fix may also well suit the codebase.

Environment

• OS: Linux 6.8 (CoreWeave node)
• Python version: 3.11.16
• FunASR version: 1.4.15, main at 486b4b7ce
• ModelScope version: 1.40.0
• PyTorch / torchaudio version: 2.11.0+cu128 / 2.11.0+cu128, transformers 5.17.0
• Install method (pip, source, Docker): source, pip install -e .
• Device (cuda, cpu, mps): cuda
• GPU model: NVIDIA B200 (183 GB), one GPU used
• CUDA/cuDNN version: 12.8 / 9.19, driver 580.126.20