`simplify_qdq` produces NaN when the folded add or concat scale overflows fp16

Aperta
  • Qualcosa non va?
#5,487 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
machine-learning, performance

Direzione di ricerca

Start with src/simplify_qdq.cpp around the scale type selection at line 408 and inspect the existing simplify_qdq tests. Add regression coverage for the reported add and concat cases, then check how the GPU or MLIR paths handle NaN through clipping and conversion. Done when the rewrite preserves the expected outputs for both cases and the tests pass.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

The quantized add rewrite in simplify_qdq folds q(add(dq(x1, s1, z1), dq(x2, s2, z2)), s, z) to q(add(dq(x1, s1/s, z1), dq(x2, s2/s, z2)), 1, z) and does the math in the common type of the scales. If all the scales are half then the ratio is computed in half as well, and it overflows to inf at about 65520. An input equal to its zero point then computes 0 * inf = NaN, and quantizelinear turns NaN into the minimum value of the output type. #5400 extends this rewrite to concat, so it has the same problem.

These are the results on the ref target with uint8 inputs and an output zero point of 20:

Case Inputs Output scale No pass After simplify_qdq
concat, half scales x1 = x2 = {7, 8, 6, 255}, s = 1, z = 7 1e-5 20 255 0 255 20 255 0 255 0 255 0 255 0 255 0 255
concat, float scales with quantize_fp16 x1 = {0, 0, 0, 0}, s1 = 1, z1 = 0; x2 = {7, 8, 6, 255}, s2 = 1e-5, z2 = 7 1e-5 20 20 20 20 20 21 19 255 0 0 0 0 20 21 19 255
add, float scales with quantize_fp16 same as above 1e-5 20 21 19 255 0 0 0 0
add, half scales x1 = {255, 255, 0}, s1 = 1, z1 = 0; x2 = {0, 255, 255}, s2 = 1, z2 = 255 1/300 20 255 20 0 255 20

Add is worse since one input at its zero point makes the whole sum NaN, no matter what the other input is. It can also fail with a finite ratio. Once 255 * ratio overflows, which is a ratio of about 257, opposite inputs compute inf - inf, as the last row shows.

I ran these on the #5400 branch. The type selection for add is the same on develop, so this has been there since #5338.

This doesnt need half scales in the model, since quantize_fp16 converts float scales to half. Also, compute_scale_zp in ORT gives an all-zero tensor a scale of 1 and a zero point of 0, so a zero-filled tensor next to a small range tensor can trigger this. It is still rare, since it needs fp16 and an output scale below about 1.5e-5 in that case. If all the branch scales are the same then quantize_fp16 merges the concat into q(dq(concat)) and the rewrite doesnt run.

I think we should use float only when 255 * ratio overflows the common type, since 255 is the largest |x - z| for 8-bit inputs. This keeps the rewrite and keeps half for normal scales. I tried this on a scratch copy of the pass. It fixes all the cases above and the existing simplify_qdq tests still pass.

The other options I tried:

  • Always use float. This changes the IR for every half scale model and add_quantized_fp16_scales would need to be updated.
  • Skip the rewrite on overflow. This lets remove_qdq_pairs drop the rounding again on every branch.

Float scales can overflow as well, but that needs scales like 1e30 and 1e-30 so I dont think its worth handling.

The fix should add regression tests for both add and concat. I only ran this on ref, so we should also check what the GPU returns when NaN goes through clip and convert, or through MLIR.

Lingua principale
C++
Stelle
338
Fork
161
Merge medio
5g 21h
PR unite (30g)
45

Preparare l'ambiente

  • Include un Dockerfile o un file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ROCm/AMDMIGraphX

Tutte le issue di ROCm/AMDMIGraphX

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.