Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Low-precision rollout normalization can overflow its divisor and erase advantages

Open Beginner friendly
  • Is something wrong?
#661 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
machine-learning

Research direction

Start with normalize_advantages_by_rollout in agentlightning/verl/per_rollout_loss.py and trace how it builds the divisor and accumulates token counts. Reproduce the float16 example from the issue, then check the listed float16, bfloat16, float32, float64, and gradient cases on CPU. Done when counts remain exact, the divisor and division use at least float32, the caller’s output dtype is preserved, and the rollout grouping and trained-row divisor are unchanged.

Written by the indexing model from the issue text.

Description

normalize_advantages_by_rollout constructs the rollout divisor in advantages.dtype. With float16 advantages, a representable normalized result can become zero because the intermediate divisor overflows. Token counts are also converted to the advantage dtype before accumulation, which rounds counts such as 257 in bfloat16.

Reproduced on main 05c7fe133175653e48c990546a3509e52dc56512, Python 3.12.2, torch 2.13.0+cpu, VERL 0.8.0, with the locked verl-cpu development environment:

import torch
from agentlightning.verl.per_rollout_loss import normalize_advantages_by_rollout

mask = torch.ones(2, 1024, dtype=torch.long)
advantages = torch.ones(2, 1024, dtype=torch.float16)
scaled = normalize_advantages_by_rollout(
    advantages, mask, ["A", "A"], num_trained_rows=64
)
print(scaled.float().sum().item())  # 0.0; expected 1 / 64 == 0.015625

The divisor is 2048 * 64 = 131072, which becomes infinity in float16, while each correctly normalized advantage (1 / 131072) is representable. This can erase the policy-gradient contribution before the loss function sees it. This is separate from reward conversion before GRPO (#639/#640): the input here is an already nonzero advantage tensor.

Expected: retain exact token counts and evaluate the existing divisor and division in at least float32, then preserve the caller's output dtype. The rollout grouping and the existing trained-row divisor stay the same. A focused patch can cover float16 overflow, bfloat16 token-count rounding, float32/float64 controls, and gradient propagation on CPU.

This is a numerical reproduction, not a model-training or GPU-quality result. Codex assisted the investigation.

Dominant language
Python
Stars
18.8k
Forks
1.7k
Avg merge
3d 13h
Merged PRs (30d)
10

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/agent-lightning

All issues in microsoft/agent-lightning

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.