`simplify_qdq` produces NaN when the folded add or concat scale overflows fp16

オープン
  • 何か問題がありますか？
#5,487 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 3 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
cpp
領域
machine-learning, performance

調査の方向性

Start with src/simplify_qdq.cpp around the scale type selection at line 408 and inspect the existing simplify_qdq tests. Add regression coverage for the reported add and concat cases, then check how the GPU or MLIR paths handle NaN through clipping and conversion. Done when the rewrite preserves the expected outputs for both cases and the tests pass.

索引モデルが issue の本文から書いたものです。

説明

bug

The quantized add rewrite in simplify_qdq folds q(add(dq(x1, s1, z1), dq(x2, s2, z2)), s, z) to q(add(dq(x1, s1/s, z1), dq(x2, s2/s, z2)), 1, z) and does the math in the common type of the scales. If all the scales are half then the ratio is computed in half as well, and it overflows to inf at about 65520. An input equal to its zero point then computes 0 * inf = NaN, and quantizelinear turns NaN into the minimum value of the output type. #5400 extends this rewrite to concat, so it has the same problem.

These are the results on the ref target with uint8 inputs and an output zero point of 20:

Case Inputs Output scale No pass After simplify_qdq
concat, half scales x1 = x2 = {7, 8, 6, 255}, s = 1, z = 7 1e-5 20 255 0 255 20 255 0 255 0 255 0 255 0 255 0 255
concat, float scales with quantize_fp16 x1 = {0, 0, 0, 0}, s1 = 1, z1 = 0; x2 = {7, 8, 6, 255}, s2 = 1e-5, z2 = 7 1e-5 20 20 20 20 20 21 19 255 0 0 0 0 20 21 19 255
add, float scales with quantize_fp16 same as above 1e-5 20 21 19 255 0 0 0 0
add, half scales x1 = {255, 255, 0}, s1 = 1, z1 = 0; x2 = {0, 255, 255}, s2 = 1, z2 = 255 1/300 20 255 20 0 255 20

Add is worse since one input at its zero point makes the whole sum NaN, no matter what the other input is. It can also fail with a finite ratio. Once 255 * ratio overflows, which is a ratio of about 257, opposite inputs compute inf - inf, as the last row shows.

I ran these on the #5400 branch. The type selection for add is the same on develop, so this has been there since #5338.

This doesnt need half scales in the model, since quantize_fp16 converts float scales to half. Also, compute_scale_zp in ORT gives an all-zero tensor a scale of 1 and a zero point of 0, so a zero-filled tensor next to a small range tensor can trigger this. It is still rare, since it needs fp16 and an output scale below about 1.5e-5 in that case. If all the branch scales are the same then quantize_fp16 merges the concat into q(dq(concat)) and the rewrite doesnt run.

I think we should use float only when 255 * ratio overflows the common type, since 255 is the largest |x - z| for 8-bit inputs. This keeps the rewrite and keeps half for normal scales. I tried this on a scratch copy of the pass. It fixes all the cases above and the existing simplify_qdq tests still pass.

The other options I tried:

  • Always use float. This changes the IR for every half scale model and add_quantized_fp16_scales would need to be updated.
  • Skip the rewrite on overflow. This lets remove_qdq_pairs drop the rounding again on every branch.

Float scales can overflow as well, but that needs scales like 1e30 and 1e-30 so I dont think its worth handling.

The fix should add regression tests for both add and concat. I only ran this on ref, so we should also check what the GPU returns when NaN goes through clip and convert, or through MLIR.

主要言語
C++
スター
338
フォーク
161
平均マージ
6日 5時間
マージ済み PR（30日）
41

環境構築

  • Dockerfile または Docker Compose ファイルあり
  • プルリクエストのテンプレートあり
  • コントリビューションガイドなし

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ROCm/AMDMIGraphX のほかの issue

ROCm/AMDMIGraphX の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。