Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Bug]: 00_load and 02_all_load validate against arange but fill the source with ones/zeros — validation can never pass

Open Beginner friendly
  • Is something wrong?
#558 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 4 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
75/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch
Domain
performance, testing

Research direction

Look at examples/00_load/load_bench.py line 253 (shmem.ones) and line 175 (torch.arange), and examples/02_all_load/all_load_bench.py line 345 (shmem.zeros) and line 213 (torch.arange). The fix is to make the validation reference match the source buffer fill: either change the fill to arange or change the validation to ones/zeros. Run the examples with --validate to see the mismatch, then adjust the test files tests/examples/test_load_bench.py and tests/examples/test_all_load_bench.py to re-enable them after the fix.

Written by the indexing model from the issue text.

Description

benchmarks bug examples iris test

Describe the bug

The load/store benchmarks validate their results against a tensor they never
wrote. expected is built with torch.arange(...), but the source buffer is
filled with shmem.ones(...) (00_load) or shmem.zeros(...) (02_all_load), so
the comparison fails by construction regardless of whether the data movement
worked.

This is the underlying cause of #119 — the tests that were skipped citing #119
(tests/examples/test_load_bench.py, tests/examples/test_all_load_bench.py)
are skipped because of this, not because of measurement accuracy.

The two symptoms are independent and both real:

  1. correctness: validation compares against the wrong reference, so
    --validate reports failure no matter what. (this issue)
  2. measurement: bandwidth is inflated because the benchmark doesn't
    subtract cached reads / use appropriate cache modifiers. (#119)

Fixing this one does not fix #119; it makes --validate mean something so
that #119 can then be worked on with a trustworthy pass/fail signal.

To Reproduce

$ python examples/00_load/load_bench.py --num_ranks 1 --validate --datatype fp32 \
    --buffer_size 536870912 --block_size 1024 --heap_size 4294967296 \
    --num_experiments 5 --num_warmup 1
INFO  [Iris] [0/1] Max absolute difference: 134217728.0

$ python examples/02_all_load/all_load_bench.py --num_ranks 1 --validate --verbose \
    --datatype fp32 --buffer_size_min 268435456 --buffer_size_max 268435456 \
    --block_size 1024 --heap_size 4294967296 --num_experiments 3 --num_warmup 1 \
    --active_ranks 1
INFO  [Iris] [0/1] Max absolute difference: 67108864.0
ERROR [Iris] [0/1] Mismatch at index (2,): C=0.0, expected=2.0
ERROR [Iris] [0/1] Validation failed.

That last line is the giveaway: index 2 contains 0.0 (the buffer was zeroed)
and is compared to 2.0 (the arange reference). The data movement may be
perfect; the reference is simply not what was stored.

Measured on ROCm 7.14 / torch 2.12.0+rocm7.14, RX 9070 XT (gfx1201).

Where

file fills source with validates against
examples/00_load/load_bench.py shmem.ones(...) (line 253) torch.arange(n_elements,...) (line 175)
examples/02_all_load/all_load_bench.py shmem.zeros(...) (line 345) torch.arange(n_elements,...) (line 213)

Compare examples/01_store/store_bench.py, which is self-consistent: its
kernel stores the index (iris.store(target_buffer + offsets, offsets, ...),
line 43), so validating against arange is correct there. That is presumably
where the arange reference was copied from.

examples/03_all_store/all_store_bench.py stores the same way and is likewise
consistent. So the bug is confined to the two *_load benchmarks.

Expected behavior

--validate passes when the data movement is correct, i.e. the reference
matches what is actually in the source buffer.

Suggested fix

Make the reference match the fill, per example. For 00_load:

source_buffer = shmem.arange(
    args["buffer_size"] // element_size_bytes, device="cuda", dtype=dtype
)

(then the existing expected = torch.arange(...) is correct), or alternatively
compare against torch.ones(...) to match the existing fill. Either is a
one-line change; I don't know which the maintainers prefer for a bandwidth
benchmark, which is why I haven't opened a patch.

Consequence worth noting

Because these two examples always fail validation, the two test files covering
them were disabled with pytest.skip(...) citing #119 — so the load and
all-load paths currently have no effective test coverage at all, and their
CI entries report as skipped rather than failed. Fixing the reference is a
prerequisite to re-enabling them.

Related

  • #119 — the measurement-accuracy issue these skips cite (separate cause)
  • I have a PR open (#555) that fixes a different example's exit code, which had
    the same "logs a failure and exits 0" shape. That is why this one stood out:
    02_all_load also exits 0 despite printing Validation failed.
Dominant language
Python
Stars
202
Forks
52
Avg merge
6d 1h
Merged PRs (30d)
18

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ROCm/iris

All issues in ROCm/iris

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.