Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

bbox_giou_reward is unbounded: reversed-corner or NaN boxes score far above 1 (RefCOCO env)

Open Beginner friendly
  • Is something wrong?
#4,871 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@jimmy-holdout is already working on this.

Since Oct 23, 2026.

  • #4872 by @jimmy-holdout — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
73/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
numpy, python
Domain
machine-learning, testing

Research direction

Start with nemo_rl/environments/rewards.py, especially bbox_giou_reward, and review the existing tests for that function; examples/prompts/refcoco.txt describes the expected box ordering. Add regression coverage for reversed, non-finite, and overflow cases, then verify malformed boxes receive zero reward and valid boxes retain their expected scores.

Written by the indexing model from the issue text.

Description

bug community-request
What happens

bbox_giou_reward parses the response box but never checks that it is well-formed. With reversed corners (x2 < x1 and y2 < y1), area_r is a positive product of two negatives, the intersection clips to 0, and when the reversed box has x1 ≥ x1_gt and x2 ≤ x2_gt (same for y), the convex hull collapses to the ground-truth box. The GIoU then works out to area_r / area_g, which can be far above 1. NaN and inf coordinates pass float() and produce NaN, inf or huge rewards, and finite coordinates large enough to overflow the areas produce NaN.

The refcoco prompt asks for [x1, y1, x2, y2] with x1, y1 top left and x2, y2 bottom right (examples/prompts/refcoco.txt), and GIoU is defined on [-1, 1], so I don't think this is intended. The existing tests only cover well-formed boxes.

Reproduce

Run repro_bbox_giou.py from the repo root. It needs only numpy and math-verify and makes no network, GPU or Ray calls:

repro_bbox_giou.py 
# Run from the repo root: python repro_bbox_giou.py  (needs numpy and math-verify; no GPU, network or Ray)
from nemo_rl.environments.rewards import bbox_giou_reward

gold = "[400, 400, 700, 700]"
cases = [
    ("gold box", "[400, 400, 700, 700]"),
    ("wrong box", "[300, 300, 600, 600]"),
    ("same wrong box, corners swapped", "[600, 600, 300, 300]"),
    ("constant [1000, 1000, 0, 0]", "[1000, 1000, 0, 0]"),
    ("constant [1e9, 1e9, -1e9, -1e9]", "[1e9, 1e9, -1e9, -1e9]"),
    ("nan", "[nan, nan, nan, nan]"),
    ("inf", "[inf, inf, -inf, -inf]"),
]
for label, box in cases:
    reward, correct = bbox_giou_reward(gold, f"<answer>{box}</answer>", giou_penalty_thres=0.5)
    print(f"{label:34s} reward={reward:.4g} correct={correct}")

Output at f0639c6 on Ubuntu 24.04 LTS (Linux 6.8), Python 3.12.3:

gold box                           reward=1 correct=True
wrong box                          reward=0.1607 correct=False
same wrong box, corners swapped    reward=1 correct=True
constant [1000, 1000, 0, 0]        reward=11.11 correct=True
constant [1e9, 1e9, -1e9, -1e9]    reward=4.444e+13 correct=True
nan                                reward=9e+07 correct=True
inf                                reward=inf correct=True

Over the RefCOCO validation split (3,811 items), with each answer wrapped in a full <think>…</think> <answer>…</answer> response and scored through VLMVerifyWorker with the env.refcoco block from vlm_grpo_3B.yaml (format 0.1, bbox_giou 0.9, giou_penalty_thres: 0.5):

answer mean env reward items with reward > 1
gold box 1.0 0 / 3811
constant [1000, 1000, 0, 0], the same for every image 7.03 (max 29.65) 3809 / 3811
constant [nan, nan, nan, nan] 1.8e8 3811 / 3811
mirror of the gold box (wrong), corners in order -0.18 0 / 3811
the same wrong mirror box with corners swapped 0.41 (1.0 on 1,304 items) 0 / 3811
Impact

In GRPO on the RefCOCO env, one image-independent answer outscores the correct box several times over, so a policy that finds the reversed-corner or NaN form is pushed towards it, and the huge values dominate group advantages and logged rewards. grpo.reward_scaling is off by default in this config, so nothing clamps it. RefCOCO is opt-in: vlm_grpo_3B.yaml trains on clevr-cogent by default and selects RefCOCO by override. So this affects runs that use the shipped env.refcoco block and RefCOCODataset, not the default run.

Possible fix

Return 0 for a response box that has non-finite coordinates or x2 <= x1 / y2 <= y1, and for a non-finite result (area overflow). That keeps the reward in [-1, 1] for a well-formed ground truth. I have a small PR with regression tests ready and will link it here.

Found while auditing public RL environments.

Dominant language
Python
Stars
2.1k
Forks
612
Avg merge
3d 14h
Merged PRs (30d)
267

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-NeMo/RL

All issues in NVIDIA-NeMo/RL

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.