huggingface/trl

Train transformer language models with reinforcement learning.

Stars: 19.3k
Forks: 3k
Open beginner issues: 17
Indexed issues: 136
Avg merge: 1d 9h
Merged PRs (30d): 191

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 18, 2026
Latest indexed: Sep 17, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

123 open issues indexed

CI rerun filter does not retry GPU allocation failures raised in a launched subprocess Open

huggingface/trl#7326 · 1 comment · 1 assignee · Opened Sep 21, 2026
Distributed deadlock and metric corruption in GRPOTrainer and RLOOTrainer when conditionally logging metrics/extra columns Open

Difficulty 4/5 3-5 days Newbie friendliness 68/100

huggingface/trl#7310 · 1 comment · Opened Sep 20, 2026
GRPO/RLOO/Distillation trainers crash on iterable datasets when dataloader_multiprocessing_context is set (default on MPS since transformers 5.15) Open

Difficulty 3/5 1-2 days Newbie friendliness 74/100

huggingface/trl#7302 · Opened Sep 19, 2026
Proposing to remove MiniLLMTrainer Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

huggingface/trl#7295 · 1 comment · Opened Sep 18, 2026
Support custom/composable advantage computation across trainers Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

huggingface/trl#7291 · Opened Sep 18, 2026
AsyncGRPO: one stale-dropped group pins the saved resume position for the rest of the run Open

Difficulty 3/5 1-2 days Newbie friendliness 74/100

huggingface/trl#7274 · Opened Sep 18, 2026
SFTTrainer's QLoRA fp16 downcast also hits QDoRA magnitude vector, causing silent no-op updates Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

huggingface/trl#7268 · 1 comment · Opened Sep 18, 2026
DPO and KTO skip DDP gradient synchronization with use_liger_kernel=True Open

huggingface/trl#7245 · 1 comment · 1 assignee · Opened Sep 16, 2026
Fused Triton kernels for the loss head Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

huggingface/trl#7244 · 1 comment · 1 reaction · 2 assignees · Opened Sep 16, 2026
router_aux_loss_coef is silently ignored on MoE configs without output_router_logits Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

huggingface/trl#7222 · 2 comments · Opened Sep 16, 2026
Apple Silicon (MPS): create_model_from_path can segfault while loading bf16 checkpoints Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

huggingface/trl#7220 · Opened Sep 15, 2026
AsyncGRPOTrainer: loss normalization across gradient accumulation and complete-rollout admission Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

huggingface/trl#7206 · Opened Sep 14, 2026
Audit the trl-internal-testing repos that have no generation script Open

huggingface/trl#7202 · 10 comments · 1 assignee · Opened Sep 14, 2026
vLLM encoder cache is not reset after weight sync Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

huggingface/trl#7198 · Opened Sep 13, 2026
Online DPO: Idefics2 image batches fail in vision tensor duplication and scoring preparation Open

Difficulty 4/5 3-5 days Newbie friendliness 54/100

huggingface/trl#7189 · Opened Sep 12, 2026
[RFC] Criteria for removing code from `trl.experimental` Open

huggingface/trl#7182 · 5 comments · 1 reaction · 1 assignee · Opened Sep 11, 2026
DPO/KTO training goes NaN with the default SDPA backend on torch 2.13 (cuDNN attention); clean with `enable_cudnn_sdp(False)` or eager Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

huggingface/trl#7181 · 1 comment · Opened Sep 11, 2026
[GRPO] Pad per minibatch instead of per generation batch Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

huggingface/trl#7179 · Opened Sep 10, 2026
Where the GPU runner hours go in the Tests workflow Open

🔧 maintenance

huggingface/trl#7169 · 3 comments · 1 assignee · Opened Sep 10, 2026
Align tiny test model configs with their reference models Open

huggingface/trl#7137 · 6 comments · 1 assignee · Opened Sep 9, 2026
Agree on an explicit criterion for raising a dependency version floor Open

huggingface/trl#7116 · 2 comments · 1 assignee · Opened Sep 8, 2026
AsyncGRPO FSDP2 weight sync hangs: interleaved full_tensor() and is_checkpoint_format=True Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

huggingface/trl#7103 · 1 reaction · Opened Sep 8, 2026
CI emits 462 `transformers` `Trainer` log warnings: "The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config" Open

huggingface/trl#7093 · 3 comments · 1 assignee · Opened Sep 7, 2026
RLOO uses higher-variance KL Estimator and should use lower variance k3 Open

Difficulty 2/5 1-3 hours Newbie friendliness 74/100

huggingface/trl#7086 · 1 comment · Opened Sep 6, 2026
[Tracking] Move off liger-kernel and make the memory-efficient loss the default Open

huggingface/trl#7063 · 6 comments · 1 assignee · Opened Sep 4, 2026
RLOO _tokenize_prompts skips VLM string-content normalization that GRPO and Distillation apply Open

Difficulty 2/5 1-3 hours Newbie friendliness 87/100

huggingface/trl#7049 · Opened Sep 4, 2026
Async trainers take an extra optimizer step when resuming at the epoch target Open

Difficulty 3/5 1-2 days Newbie friendliness 74/100

huggingface/trl#7035 · Opened Sep 3, 2026
Bug: CLI --accelerate_config does not override YAML config Open

Difficulty 3/5 1-2 days Newbie friendliness 76/100

huggingface/trl#7031 · Opened Sep 3, 2026
create_model_from_path ignores subfolder when inferring the model architecture Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

huggingface/trl#7025 · Opened Sep 3, 2026
Replace the introspection guard in `sft_qwen3_8b_1m_context.py` with a plain version comparison Open

huggingface/trl#7021 · 1 assignee · Opened Sep 2, 2026
get_repetition_penalty_reward accepts non-positive ngram_size Open

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

huggingface/trl#7015 · 3 comments · Opened Sep 2, 2026
GRPOTrainer silently drops the MoE auxiliary loss when use_liger_kernel=True Open

Difficulty 2/5 1-3 hours Newbie friendliness 85/100

huggingface/trl#7009 · 3 comments · Opened Sep 2, 2026
create_model_from_path crashes when config lacks architecture metadata Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

huggingface/trl#7006 · Opened Sep 2, 2026
🐛 [Bug] get_dataset raises ZeroDivisionError for zero-sum fractions Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

huggingface/trl#6981 · 1 comment · Opened Aug 31, 2026
Online DPO vLLM server fails on mixed image and text-only batches Open

Difficulty 3/5 1-2 days Newbie friendliness 76/100

huggingface/trl#6967 · Opened Aug 29, 2026
`think_format_reward` always returns 0.0 for models whose chat template prefills `<think>` Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

huggingface/trl#6966 · 2 comments · Opened Aug 28, 2026
AsyncGRPO has no importance-sampling correction, but TRL still imposes --logprobs-mode processed_logprobs on its server Open

Difficulty 5/5 Over a week Newbie friendliness 48/100

huggingface/trl#6945 · 3 comments · Opened Aug 27, 2026
prepare_multimodal_messages_vllm fails on content-less tool-call messages Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

huggingface/trl#6928 · 1 comment · Opened Aug 26, 2026
CI rerun filter does not retry GPU allocation failures wrapped in another exception Open

huggingface/trl#6917 · 4 comments · 1 assignee · Opened Aug 25, 2026
Bug: unpair IterableDatasetDict preference datasets Open

Difficulty 2/5 1-3 hours Newbie friendliness 74/100

huggingface/trl#6877 · 1 comment · Opened Aug 23, 2026
Bug: normalize --accelerate_config=<name> CLI syntax Open

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

huggingface/trl#6874 · Opened Aug 23, 2026
Tiny test models are incompatible with Flash Attention (head_size=2) Open

huggingface/trl#6857 · 1 comment · 1 reaction · 1 assignee · Opened Aug 21, 2026
CI fails for slow tests on multiple GPUs: Fatal Python error: Segmentation fault Open

huggingface/trl#6836 · 1 comment · 1 assignee · Opened Aug 20, 2026
RFC: Add single-image vision-language model support to A2POTrainer Open

Difficulty 5/5 Over a week Newbie friendliness 38/100

huggingface/trl#6833 · 1 comment · 1 reaction · Opened Aug 20, 2026
RFC: Add end-to-end vision-language model support to SDPOTrainer Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

huggingface/trl#6832 · 1 comment · Opened Aug 20, 2026
RFC: Support A2PO with DeepSpeed ZeRO-3 via an explicit reference model Open

Difficulty 4/5 3-5 days Newbie friendliness 46/100

huggingface/trl#6831 · 1 comment · Opened Aug 20, 2026
SDPO PEFT EMA initialization reads partitioned state under ZeRO-3 Open

Difficulty 4/5 3-5 days Newbie friendliness 50/100

huggingface/trl#6828 · Opened Aug 20, 2026
Metrics-only inaccuracies in RL trainers: unmasked stats (OnlineDPO, PPO), wrong XPO logps metric, GRPO policy_loss only logged with entropy bonus Open

Difficulty 4/5 3-5 days Newbie friendliness 68/100

huggingface/trl#6809 · Opened Aug 19, 2026
GMPO: use_liger_kernel=True silently bypasses the GMPO loss; MoE aux loss and entropy bonus dropped (gaps not covered by #6095) Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

huggingface/trl#6808 · 2 comments · Opened Aug 19, 2026
gspo_token: _compute_loss is a stale copy of GRPO's — use_bias_correction_kl, OPSM, entropy bonus, and MoE aux loss silently ignored Open

Difficulty 4/5 3-5 days Newbie friendliness 68/100

huggingface/trl#6807 · 1 comment · Opened Aug 19, 2026

A short digest of beginner-friendly GitHub issues.