[Bug]: 00_load and 02_all_load validate against arange but fill the source with ones/zeros — validation can never pass

オープン 初心者向け
  • 何か問題がありますか？
#558 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
2/5
見積もり時間
1〜3時間
初心者へのやさしさ
75/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python, pytorch
領域
performance, testing

調査の方向性

examples/00_load/load_bench.py の 253 行目 (shmem.ones) と 175 行目 (torch.arange)、および examples/02_all_load/all_load_bench.py の 345 行目 (shmem.zeros) と 213 行目 (torch.arange) を確認してください。修正は、検証参照をソースバッファのフィルと一致させることです：フィルを arange に変更するか、検証を ones/zeros に変更します。--validate を付けて例を実行して不一致を確認し、修正後にテストファイル tests/examples/test_load_bench.py と tests/examples/test_all_load_bench.py を調整して再有効化してください。

索引モデルが issue の本文から書いたものです。

説明

benchmarks bug examples iris test

Describe the bug

The load/store benchmarks validate their results against a tensor they never
wrote. expected is built with torch.arange(...), but the source buffer is
filled with shmem.ones(...) (00_load) or shmem.zeros(...) (02_all_load), so
the comparison fails by construction regardless of whether the data movement
worked.

This is the underlying cause of #119 — the tests that were skipped citing #119
(tests/examples/test_load_bench.py, tests/examples/test_all_load_bench.py)
are skipped because of this, not because of measurement accuracy.

The two symptoms are independent and both real:

  1. correctness: validation compares against the wrong reference, so
    --validate reports failure no matter what. (this issue)
  2. measurement: bandwidth is inflated because the benchmark doesn't
    subtract cached reads / use appropriate cache modifiers. (#119)

Fixing this one does not fix #119; it makes --validate mean something so
that #119 can then be worked on with a trustworthy pass/fail signal.

To Reproduce

$ python examples/00_load/load_bench.py --num_ranks 1 --validate --datatype fp32 \
    --buffer_size 536870912 --block_size 1024 --heap_size 4294967296 \
    --num_experiments 5 --num_warmup 1
INFO  [Iris] [0/1] Max absolute difference: 134217728.0

$ python examples/02_all_load/all_load_bench.py --num_ranks 1 --validate --verbose \
    --datatype fp32 --buffer_size_min 268435456 --buffer_size_max 268435456 \
    --block_size 1024 --heap_size 4294967296 --num_experiments 3 --num_warmup 1 \
    --active_ranks 1
INFO  [Iris] [0/1] Max absolute difference: 67108864.0
ERROR [Iris] [0/1] Mismatch at index (2,): C=0.0, expected=2.0
ERROR [Iris] [0/1] Validation failed.

That last line is the giveaway: index 2 contains 0.0 (the buffer was zeroed)
and is compared to 2.0 (the arange reference). The data movement may be
perfect; the reference is simply not what was stored.

Measured on ROCm 7.14 / torch 2.12.0+rocm7.14, RX 9070 XT (gfx1201).

Where

file fills source with validates against
examples/00_load/load_bench.py shmem.ones(...) (line 253) torch.arange(n_elements,...) (line 175)
examples/02_all_load/all_load_bench.py shmem.zeros(...) (line 345) torch.arange(n_elements,...) (line 213)

Compare examples/01_store/store_bench.py, which is self-consistent: its
kernel stores the index (iris.store(target_buffer + offsets, offsets, ...),
line 43), so validating against arange is correct there. That is presumably
where the arange reference was copied from.

examples/03_all_store/all_store_bench.py stores the same way and is likewise
consistent. So the bug is confined to the two *_load benchmarks.

Expected behavior

--validate passes when the data movement is correct, i.e. the reference
matches what is actually in the source buffer.

Suggested fix

Make the reference match the fill, per example. For 00_load:

source_buffer = shmem.arange(
    args["buffer_size"] // element_size_bytes, device="cuda", dtype=dtype
)

(then the existing expected = torch.arange(...) is correct), or alternatively
compare against torch.ones(...) to match the existing fill. Either is a
one-line change; I don't know which the maintainers prefer for a bandwidth
benchmark, which is why I haven't opened a patch.

Consequence worth noting

Because these two examples always fail validation, the two test files covering
them were disabled with pytest.skip(...) citing #119 — so the load and
all-load paths currently have no effective test coverage at all, and their
CI entries report as skipped rather than failed. Fixing the reference is a
prerequisite to re-enabling them.

Related

  • #119 — the measurement-accuracy issue these skips cite (separate cause)
  • I have a PR open (#555) that fixes a different example's exit code, which had
    the same "logs a failure and exits 0" shape. That is why this one stood out:
    02_all_load also exits 0 despite printing Validation failed.
主要言語
Python
スター
203
フォーク
57
平均マージ
3日 7時間
マージ済み PR（30日）
31

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ROCm/iris のほかの issue

ROCm/iris の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。