Question: can I take INT4-AWQ for llama and FP8 for the stable_diffusion SDXL-Turbo UNet?

Aperta
  • Qualcosa non va?
#1,928 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
python, pytorch
Ambito
machine-learning

Direzione di ricerca

This issue asks for maintainer guidance rather than requesting a change, so wait for a decision on whether either proposal is available before starting. If approved, read the qwen3_8 _AWQFormat and load_qwen3_8_awq path for llama, or #1907 and families/stable_diffusion/tests/manifests/sdxl-turbo.json for SDXL-Turbo; completion criteria still need maintainer confirmation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

question
Topic

Architecture or contribution guidance

Question

@zhenshanx-nv I'd like to contribute two quantization paths that follow your self-quantization series (#1907, #1898, #1918). Could you confirm both are free to take, so I stay out of your queue?

1. llama: native INT4-AWQ (W4A16), first

llama accepts fp8 and nvfp4 today. Native INT4-AWQ exists in qwen3_8, qwen3_5, internvl and phi4_multimodal, all loading prepared ModelOpt W4A16_AWQ checkpoints. llama has no public W4A16 checkpoint, so I'd self-quantize the original checkpoint at build time:

  • calibrate_awq_scales.py (developer tool, ModelOpt INT4_AWQ_CFG) produces the per-layer pre_quant_scale, checked in and keyed to the checkpoint, like fp8_activation_scales.json.
  • The build computes INT4 weights and block scales (block size 128) from the original weights and emits INT4 constant → Dequantize → MatMul with the smoothing multiply on the activation, matching _AWQFormat in qwen3_8.
  • Decoder q/k/v/o and MLP projections are quantized; embeddings, norms and lm_head stay FP16. A missing scale fails the build.
  • E2E case on nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1, already pinned at d552708a in the llama manifests.

2. stable_diffusion: FP8 for the SDXL-Turbo UNet, after #1909 lands

  • Same structure as #1907: quantization.py, calibrate_fp8_scales.py, checked-in scales, fp16 Dequantize.
  • UNet attention and feed-forward linears are quantized; convolutions, norms, time embedding, text encoders and VAE stay FP16.

Questions

  1. Are both free to take? If either is already on your list, which family would you like an external contributor to pick up instead?
  2. For INT4-AWQ, is a checked-in pre_quant_scale file (a few MB for a 4B model) acceptable, or do you prefer calibration at build time?
  3. I validate on an RTX 3090 (SM86) and an RTX 6000 Ada (SM89) with TensorRT 11.1: quantized GEMMs confirmed in the plan inspector, engine size and latency against the fp16 engine, and output parity against the unquantized reference. Does that cover review without B200/B300 numbers?
What have you tried?

Read #1907, #1898, #1918 and the qwen3_8 INT4-AWQ path (_AWQFormat, load_qwen3_8_awq). No open PR or issue covers llama INT4-AWQ or stable_diffusion quantization.

TensorRT-Model-Connect version

main @ 2c2256cc5ac5840d933483d6ee8dc8083fc78c95

Model and revision

nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 @ d552708a9d575fa8d4a690b988fd870d65279f98; stabilityai/sdxl-turbo at the revision pinned in families/stable_diffusion/tests/manifests/sdxl-turbo.json

Environment details

Ubuntu (Linux 6.8), RTX 3090 (SM86) + RTX 6000 Ada (SM89), driver 580.178.04, TensorRT 11.1.0.106, source build

Submission checks
  • I searched the documentation and open and closed issues first.
  • This question contains no secrets, private URLs, or restricted artifacts.
Lingua principale
Python
Stelle
283
Fork
75
Merge medio
1g 17h
PR unite (30g)
218

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/TensorRT-Model-Connect

Tutte le issue di NVIDIA/TensorRT-Model-Connect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.