SFT truncation filter retains samples whose only unmasked label is removed by causal shifting

Open Beginner friendly
  • Is something wrong?
#8,296 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
74/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
machine-learning

Research direction

Start at the SFT preprocessing filter that checks any(label != -100 for label in example["labels"]) after truncation, and compare it with the causal shift that drops position zero (labels[..., 1:]). Run the reproduction script in the issue to confirm that a sample with labels [1, -100, -100] is kept with zero usable targets. Done means such a sample is dropped, or the filter counts only labels that survive the shift, and a regression test covers the case.

Written by the indexing model from the issue text.

Description

Reproduction cc @albertvillanova @qgallouedec

Related prior reports: #6668 and #6671.

This reproduces with preprocessing enabled and packing=False. Unlike a sample whose labels are entirely -100, one unmasked label remains at position zero. The existing filter keeps the sample, although causal shifting removes its only unmasked label.

In a fresh Python 3.11 environment:

python -m pip install \
  "trl @ git+https://github.com/huggingface/trl.git@0bc32146803125345102fb09af154b96e2989485" \
  "torch==2.6.0" "transformers==4.57.6" \
  "accelerate==1.15.0" "datasets==5.0.1"

import torch
from datasets import Dataset
from trl import SFTConfig, SFTTrainer

torch.manual_seed(0)
data = Dataset.from_list([
    {"input_ids": [1, 2, 3, 4], "labels": [1, -100, -100, 4]}
])

trainer = SFTTrainer(
    model="trl-internal-testing/tiny-Qwen2ForCausalLM-2.5",
    train_dataset=data,
    args=SFTConfig(
        output_dir="/tmp/trl-first-label-repro",
        max_length=3,
        packing=False,
        max_steps=1,
        per_device_train_batch_size=1,
        use_cpu=True,
        bf16=False,
        fp16=False,
        weight_decay=0.0,
        report_to="none",
        save_strategy="no",
        disable_tqdm=True,
    ),
)

labels = trainer.train_dataset[0]["labels"]
print("rows_after_preparation:", len(trainer.train_dataset))
print("labels_after_truncation:", labels)
print("non_ignored_labels:", sum(label != -100 for label in labels))
print("usable_next_token_targets:", sum(label != -100 for label in labels[1:]))

before = {
    name: parameter.detach().clone()
    for name, parameter in trainer.model.named_parameters()
    if parameter.requires_grad
}

result = trainer.train()

unchanged = all(
    torch.equal(before[name], parameter.detach())
    for name, parameter in trainer.model.named_parameters()
    if parameter.requires_grad
)

print("training_loss:", result.training_loss)
print("all_trainable_parameters_unchanged:", unchanged)

Observed output (excluding progress bars):

rows_after_preparation: 1
labels_after_truncation: [1, -100, -100]
non_ignored_labels: 1
usable_next_token_targets: 0
training_loss: 0.0
all_trainable_parameters_unchanged: True

Before truncation, position three contains a usable target. Truncation removes it, leaving only the position-zero label. The filter checks all labels (any(label != -100 for label in example["labels"])), whereas the causal loss shifts targets via labels[..., 1:].

How did you hit this?

I found this during source analysis and regression testing of completion only masking, then confirmed it with the CPU training run above.

The reproduction uses the official tiny Qwen2.5 model and pre tokenized inputs with custom labels. For users with custom label SFT datasets, truncation can remove every usable target while the position zero label keeps the sample in the dataset. Training then completes zero loss steps without updating parameters.

I have a local fix and regression tests and would like to contribute. Please let me know if this case is in scope and if you would like me to submit a PR.

System Info

Platform: macOS 26.5.2, arm64
Python: 3.11.14
TRL: 1.18.0.dev0
Upstream commit: 0bc32146803125345102fb09af154b96e2989485 (main)
PyTorch: 2.6.0
Transformers: 4.57.6
Tokenizers: 0.22.2
Accelerate: 1.15.0
Datasets: 5.0.1
Reproduction device: CPU

Checklist
  • I have checked that my issue isn't already filed (see open issues)
  • I have included my system information
  • Any code provided is minimal, complete, and reproducible (more on MREs)
  • Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
  • Any traceback provided is complete
Dominant language
Python
Stars
19.7k
Forks
3.1k
Avg merge
2d 13h
Merged PRs (30d)
283

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/trl

All issues in huggingface/trl

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.