CUDA INT8 fake quantization converts channels-last inputs to default contiguous layout

Aperta
  • Qualcosa non va?
#3,195 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@thuiaaa ci sta già lavorando.

Dal 3/11/2026.

  • #3196 di @thuiaaa — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
50/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp, python
Ambito
machine-learning

Direzione di ricerca

Start with the CUDA fake-quant entry points that call inputs.contiguous() without a memory format, for both per-tensor and per-channel paths, and with the in-place check that rejects channels-last inputs. Run the reproducer in the issue to confirm the output stride changes from (105, 1, 21, 3). Done means the layout is preserved for contiguous_format, channels_last and channels_last_3d, while quantized values stay identical and the CUDA extension tests pass.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

CUDA INT8 fake quantization converts channels-last inputs to the default contiguous memory format. The output has the correct quantized values, but loses the input memory layout.

The main motivation is downstream numerical alignment. In practical use, we observed a substantial downstream numerical mismatch despite identical fake-quant values; preserving the output memory format restored alignment with the layout-preserving reference in that case. The conversion also adds an unnecessary full-tensor copy. The per-tensor and per-channel CUDA entry points both call inputs.contiguous() without a memory format.

Steps/Code to reproduce bug

Run on an unmodified checkout of NVIDIA/Model-Optimizer at 472ecb39ad498a6b0d818583d5b1dc483a708abf with CUDA and the ModelOpt CUDA extension available:

import torch
from modelopt.torch.quantization.extensions import get_cuda_ext

ext = get_cuda_ext(raise_if_failed=True)
x = torch.linspace(-1, 1, 210, device="cuda").reshape(2, 3, 5, 7)
x = x.contiguous(memory_format=torch.channels_last)
amax = torch.tensor(1.0, device="cuda")

y = ext.fake_tensor_quant(x, amax)
reference = ext.fake_tensor_quant(x.contiguous(), amax)

print("input stride:", x.stride())
print("output stride:", y.stride())
print("same quantized values:", torch.equal(y, reference))
print("channels-last output:", y.is_contiguous(memory_format=torch.channels_last))

Observed on the original CUDA implementation:

input stride: (105, 1, 21, 3)
output stride: (105, 35, 7, 1)
same quantized values: True
channels-last output: False

This synthetic reproducer demonstrates the layout side effect. The downstream numerical mismatch described above is an application observation, rather than a result measured by this minimal reproducer.

The public tensor_quant.fake_tensor_quant API reaches the same CUDA implementation. Per-channel quantization through fake_tensor_quant_with_axis and 5D channels_last_3d inputs also lose their memory format. The in-place per-tensor entry point rejects channels-last inputs despite their dense contiguous storage.

Expected behavior

Preserve supported contiguous input memory formats (contiguous_format, channels_last, channels_last_3d) without changing the quantization formula, ranges, or output values. Continue to handle non-contiguous inputs safely.

A possible localized fix is to prepare out-of-place inputs with inputs.contiguous(inputs.suggest_memory_format()) and accept the same contiguous formats in the in-place check. The per-channel kernel already uses the actual axis stride to select its amax. A candidate fix and synthetic regression tests have been prepared.

Verification

The reproducer above was executed against both original and fixed CUDA extensions with PyTorch 2.12.1 / CUDA 13.0. The candidate preserves (105, 1, 21, 3) while retaining identical quantized values.

Synthetic regression tests cover per-tensor, every per-channel axis, in-place behavior, non-contiguous inputs, and FP32/FP16/BF16. The original extension fails 12 channels-last cases; all 76 CUDA extension tests and 47 CPU tensor-quantization tests pass with the candidate.

Who can help?

Quantization maintainers.

System information

  • OS: Ubuntu 22.04.5 LTS (container).
  • CPU architecture: x86_64.
  • GPU: one NVIDIA GeForce RTX 5090, approximately 32 GB.
  • Python: 3.12.3.
  • ModelOpt: upstream commit 472ecb39ad498a6b0d818583d5b1dc483a708abf.
  • CUDA toolkit: 13.0 (V13.0.88); driver: 595.71.05.
  • PyTorch: 2.12.1+cu130 (within the declared PyTorch >=2.10 range).
  • Also reproduced with PyTorch 2.7.0+cu128 / CUDA 12.8 as supplemental older-version evidence.
  • Transformers, TensorRT-LLM, ONNXRuntime, TensorRT: not required by the reproducer.
Lingua principale
Python
Stelle
5.5k
Fork
800
Merge medio
3g 12h
PR unite (30g)
278

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/Model-Optimizer

Tutte le issue di NVIDIA/Model-Optimizer

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.