`simplify_qdq` produces NaN when the folded add or concat scale overflows fp16

Abierto
  • ¿Algo no está bien?
#5,487 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
cpp
Área
machine-learning, performance

Línea de trabajo

Start with src/simplify_qdq.cpp around the scale type selection at line 408 and inspect the existing simplify_qdq tests. Add regression coverage for the reported add and concat cases, then check how the GPU or MLIR paths handle NaN through clipping and conversion. Done when the rewrite preserves the expected outputs for both cases and the tests pass.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug

The quantized add rewrite in simplify_qdq folds q(add(dq(x1, s1, z1), dq(x2, s2, z2)), s, z) to q(add(dq(x1, s1/s, z1), dq(x2, s2/s, z2)), 1, z) and does the math in the common type of the scales. If all the scales are half then the ratio is computed in half as well, and it overflows to inf at about 65520. An input equal to its zero point then computes 0 * inf = NaN, and quantizelinear turns NaN into the minimum value of the output type. #5400 extends this rewrite to concat, so it has the same problem.

These are the results on the ref target with uint8 inputs and an output zero point of 20:

Case Inputs Output scale No pass After simplify_qdq
concat, half scales x1 = x2 = {7, 8, 6, 255}, s = 1, z = 7 1e-5 20 255 0 255 20 255 0 255 0 255 0 255 0 255 0 255
concat, float scales with quantize_fp16 x1 = {0, 0, 0, 0}, s1 = 1, z1 = 0; x2 = {7, 8, 6, 255}, s2 = 1e-5, z2 = 7 1e-5 20 20 20 20 20 21 19 255 0 0 0 0 20 21 19 255
add, float scales with quantize_fp16 same as above 1e-5 20 21 19 255 0 0 0 0
add, half scales x1 = {255, 255, 0}, s1 = 1, z1 = 0; x2 = {0, 255, 255}, s2 = 1, z2 = 255 1/300 20 255 20 0 255 20

Add is worse since one input at its zero point makes the whole sum NaN, no matter what the other input is. It can also fail with a finite ratio. Once 255 * ratio overflows, which is a ratio of about 257, opposite inputs compute inf - inf, as the last row shows.

I ran these on the #5400 branch. The type selection for add is the same on develop, so this has been there since #5338.

This doesnt need half scales in the model, since quantize_fp16 converts float scales to half. Also, compute_scale_zp in ORT gives an all-zero tensor a scale of 1 and a zero point of 0, so a zero-filled tensor next to a small range tensor can trigger this. It is still rare, since it needs fp16 and an output scale below about 1.5e-5 in that case. If all the branch scales are the same then quantize_fp16 merges the concat into q(dq(concat)) and the rewrite doesnt run.

I think we should use float only when 255 * ratio overflows the common type, since 255 is the largest |x - z| for 8-bit inputs. This keeps the rewrite and keeps half for normal scales. I tried this on a scratch copy of the pass. It fixes all the cases above and the existing simplify_qdq tests still pass.

The other options I tried:

  • Always use float. This changes the IR for every half scale model and add_quantized_fp16_scales would need to be updated.
  • Skip the rewrite on overflow. This lets remove_qdq_pairs drop the rounding again on every branch.

Float scales can overflow as well, but that needs scales like 1e30 and 1e-30 so I dont think its worth handling.

The fix should add regression tests for both add and concat. I only ran this on ref, so we should also check what the GPU returns when NaN goes through clip and convert, or through MLIR.

Lenguaje dominante
C++
Estrellas
338
Forks
161
Merge medio
6 d 5 h
PR fusionados (30 d)
41

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ROCm/AMDMIGraphX

Todos los issues de ROCm/AMDMIGraphX

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.