CUDA INT8 fake quantization converts channels-last inputs to default contiguous layout

Abierto
  • ¿Algo no está bien?
#3,195 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

@thuiaaa ya está trabajando en esto.

Desde el 3/11/2026.

  • #3196 de @thuiaaa — abierto

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
50/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
cpp, python
Área
machine-learning

Línea de trabajo

Start with the CUDA fake-quant entry points that call inputs.contiguous() without a memory format, for both per-tensor and per-channel paths, and with the in-place check that rejects channels-last inputs. Run the reproducer in the issue to confirm the output stride changes from (105, 1, 21, 3). Done means the layout is preserved for contiguous_format, channels_last and channels_last_3d, while quantized values stay identical and the CUDA extension tests pass.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

CUDA INT8 fake quantization converts channels-last inputs to the default contiguous memory format. The output has the correct quantized values, but loses the input memory layout.

The main motivation is downstream numerical alignment. In practical use, we observed a substantial downstream numerical mismatch despite identical fake-quant values; preserving the output memory format restored alignment with the layout-preserving reference in that case. The conversion also adds an unnecessary full-tensor copy. The per-tensor and per-channel CUDA entry points both call inputs.contiguous() without a memory format.

Steps/Code to reproduce bug

Run on an unmodified checkout of NVIDIA/Model-Optimizer at 472ecb39ad498a6b0d818583d5b1dc483a708abf with CUDA and the ModelOpt CUDA extension available:

import torch
from modelopt.torch.quantization.extensions import get_cuda_ext

ext = get_cuda_ext(raise_if_failed=True)
x = torch.linspace(-1, 1, 210, device="cuda").reshape(2, 3, 5, 7)
x = x.contiguous(memory_format=torch.channels_last)
amax = torch.tensor(1.0, device="cuda")

y = ext.fake_tensor_quant(x, amax)
reference = ext.fake_tensor_quant(x.contiguous(), amax)

print("input stride:", x.stride())
print("output stride:", y.stride())
print("same quantized values:", torch.equal(y, reference))
print("channels-last output:", y.is_contiguous(memory_format=torch.channels_last))

Observed on the original CUDA implementation:

input stride: (105, 1, 21, 3)
output stride: (105, 35, 7, 1)
same quantized values: True
channels-last output: False

This synthetic reproducer demonstrates the layout side effect. The downstream numerical mismatch described above is an application observation, rather than a result measured by this minimal reproducer.

The public tensor_quant.fake_tensor_quant API reaches the same CUDA implementation. Per-channel quantization through fake_tensor_quant_with_axis and 5D channels_last_3d inputs also lose their memory format. The in-place per-tensor entry point rejects channels-last inputs despite their dense contiguous storage.

Expected behavior

Preserve supported contiguous input memory formats (contiguous_format, channels_last, channels_last_3d) without changing the quantization formula, ranges, or output values. Continue to handle non-contiguous inputs safely.

A possible localized fix is to prepare out-of-place inputs with inputs.contiguous(inputs.suggest_memory_format()) and accept the same contiguous formats in the in-place check. The per-channel kernel already uses the actual axis stride to select its amax. A candidate fix and synthetic regression tests have been prepared.

Verification

The reproducer above was executed against both original and fixed CUDA extensions with PyTorch 2.12.1 / CUDA 13.0. The candidate preserves (105, 1, 21, 3) while retaining identical quantized values.

Synthetic regression tests cover per-tensor, every per-channel axis, in-place behavior, non-contiguous inputs, and FP32/FP16/BF16. The original extension fails 12 channels-last cases; all 76 CUDA extension tests and 47 CPU tensor-quantization tests pass with the candidate.

Who can help?

Quantization maintainers.

System information

  • OS: Ubuntu 22.04.5 LTS (container).
  • CPU architecture: x86_64.
  • GPU: one NVIDIA GeForce RTX 5090, approximately 32 GB.
  • Python: 3.12.3.
  • ModelOpt: upstream commit 472ecb39ad498a6b0d818583d5b1dc483a708abf.
  • CUDA toolkit: 13.0 (V13.0.88); driver: 595.71.05.
  • PyTorch: 2.12.1+cu130 (within the declared PyTorch >=2.10 range).
  • Also reproduced with PyTorch 2.7.0+cu128 / CUDA 12.8 as supplemental older-version evidence.
  • Transformers, TensorRT-LLM, ONNXRuntime, TensorRT: not required by the reproducer.
Lenguaje dominante
Python
Estrellas
5.5k
Forks
800
Merge medio
3 d 12 h
PR fusionados (30 d)
278

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/Model-Optimizer

Todos los issues de NVIDIA/Model-Optimizer

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.