TDT Head Stagnation in parakeet-tdt_ctc-110m Fine-tuning on Persian Data

[update 2025-07-07]: I initialized a new run with batch_size=8 and accumulate_grad_batches=128 as mentioned by @MahmoudAshraf97 [update 2025-07-07]: I stopped the training to pull the new changes and apply the second modification mentioned by @jeremy110

TODO

• set the ctc-weight to zero to see how the TDT head does by itself.

@nithinraok

Description of Issue:

I am fine-tuning the parakeet-tdt_ctc-110m ASR model on approximately 1800 hours of Persian speech data, following the speech_to_text_finetune.py recipe. I've also trained a 1024-token BPE tokenizer on the training data and updated the model configuration to use it.

After 9 epochs of training, I've observed that while the CTC head's performance is improving steadily (as indicated by train_ctc_loss, training_batch_wer_ctc, and val_wer_ctc), the TDT head shows little to no improvement, with its WER remaining high (visible when comparing training_batch_wer and val_wer against their CTC counterparts).

Another issue is that the predicted text in the logs are generating the token "⁇", example: [NeMo I 2025-07-07 02:43:21 wer:330] reference:درست مانند منشور که نور را به طیفی از رنگ‌های مختلف تقسیم می‌کند. [NeMo I 2025-07-07 02:43:21 wer:331] predicted: ⁇ راست ماننسور که نور را شریر‌ مختلف، تقس می

Note that this is happening in the predicted sentences not the references, ruling out the possibility of out of vocab tokens in the reference. This happens rarely though. CTC head used to make this token at first but now there is no sign. Maybe there is something wrong with the inference code?!

Below is the relevant portion of my train_config.yaml with paths replaced for privacy:

name: "parakeet_tdt_ctc_110m_v41_punc/2025-07-04_18-02-09"

# Initialize from your local .nemo instead of HF
init_from_nemo_model: "parakeet-tdt_ctc-110m.nemo"
init_from_pretrained_model: null

model:
  sample_rate: 16000

  train_ds:
    manifest_filepath: "ASR/Datasets/Final_Train_Normalized_Punc.json"
    sample_rate: ${model.sample_rate}
    batch_size: 6 # you may increase batch_size if your memory allows
    shuffle: true
    num_workers: 8
    pin_memory: true
    max_duration: 30
    min_duration: null
    channel_selector: "average"  # Averages stereo channels to mono
    # tarred datasets
    is_tarred: false
    tarred_audio_filepaths: null
    shuffle_n: 2048
    # bucketing params
    bucketing_strategy: "fully_randomized"
    bucketing_batch_size: null

  validation_ds:
    manifest_filepath: "ASR/Datasets/Final_Valid_Normalized_Punc.json"
    sample_rate: ${model.sample_rate}
    batch_size: 32
    max_duration: 30
    min_duration: null
    channel_selector: "average"  # Averages stereo channels to mono
    shuffle: false
    use_start_end_token: false
    num_workers: 8
    pin_memory: true

  char_labels: # use for char based models
  update_labels: false
  labels: null # example list config: \[' ', 'a', 'b', 'c'\]

  # Swap in your Persian BPE tokenizer under model
  tokenizer:
    update_tokenizer: true
    dir: "ASR/Tokenizers/V41_Nemo_Tokenizer_1024/tokenizer_spe_bpe_v1024_max_4"
    type: bpe
  # SpecAugment (unchanged)
  spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 2
    time_masks: 10
    freq_width: 27
    time_width: 0.05

  # ORIGINAL optimizer & scheduler from speech_to_text_finetune.yaml
  optim:
    name: adamw
    lr: 1e-4
    betas: [0.9, 0.98]
    weight_decay: 1e-3
    sched:
      name: CosineAnnealing
      warmup_steps: 5000
      warmup_ratio: null
      min_lr: 5e-6

trainer:
  devices: -1
  num_nodes: 1
  max_epochs: 50
  max_steps: -1 # computed at runtime if not set
  val_check_interval: 1.0 # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
  accelerator: auto
  strategy:
    _target_: lightning.pytorch.strategies.DDPStrategy
    gradient_as_bucket_view: true
  accumulate_grad_batches: 1
  gradient_clip_val: 0.0
  precision: 32  # Use mixed precision # 16, 32, or bf16
  log_every_n_steps: 10  # Interval of logging.
  enable_progress_bar: True
  num_sanity_val_steps: 0 # number of steps to perform validation steps for sanity check the validation process before starting the training, setting to 0 disables it
  check_val_every_n_epoch: 1 # number of evaluations on validation every n epochs
  sync_batchnorm: true
  enable_checkpointing: False  # Provided by exp_manager
  logger: false  # Provided by exp_manager
  benchmark: false # needs to be false for models with variable-length speech input as it slows down training

exp_manager:
  exp_dir: "ASR/exp"
  name: ${name}
  create_tensorboard_logger: true
  create_checkpoint_callback: true
  checkpoint_callback_params:
    monitor: "val_wer"              # Use epoch as a dummy monitor metric
    mode: "min"                   # Higher epoch is better
    every_n_epochs: 1              # Save every epoch
    save_top_k: -1                 # Save all checkpoints
    save_last: true              # Do not only keep a "last" file
    save_on_train_epoch_end: True  # Save after validation
    always_save_nemo: true      # Save .nemo file each time
    filename: "model-{epoch:02d}-{val_wer:.2f}"  # Unique filename with epoch & metric
  
  resume_if_exists: true
  resume_ignore_no_checkpoint: false

  create_wandb_logger: false
  wandb_logger_kwargs:
    name: null
    project: null

(Plots will be attached separately, showing trends for epochs, train_loss, train_rnnt_loss, train_ctc_loss, training_batch_wer, training_batch_wer_ctc, val_wer, and val_wer_ctc.)

Possible Reasons for TDT Stagnation:

1. Tokenizer Mismatch/Optimization for TDT: While a new tokenizer is used, the TDT head's prediction network might not be effectively adapting to the new token set or the specific characteristics of Persian language (e.g., common sequences, subword units) in the same way the CTC head does.

2. Learning Rate and Schedule: The current learning rate (1e-4) and CosineAnnealing schedule with 5000 warmup steps might not be optimal for the TDT head to sufficiently adapt, especially given the new language and tokenizer. TDT often requires more careful tuning of these parameters.

System Configuration

• Operating System: Ubuntu 22.04.1 LTS
• Kernel: Linux eri4090 6.8.0-60-generic x86_64
• GPU: NVIDIA GeForce RTX 4090
• NVIDIA Driver Version: 560.35.03
• CUDA Version: 12.6
• GPU Memory: 24 GB
• NeMo Toolkit Version: 2.4.0rc0
• Torch Version: 2.6.0+cu126

I appreciate any insights or suggestions to resolve this issue.