Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Nvidia Modelopt structured 2:4 weight sparsity showing no speed improvement compared to dense model

Aperta
  • Qualcosa non va?
#1,974 6 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@kaix-nv ci sta già lavorando.

Dal 14/7/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug

Describe the bug

A Qwen3-4B-Instruct-2507 hugging face model sparsified to 2:4 structured sparsity[Weight sparsity] with ModelOpt
(SparseGPT post-training sparsification + sparsity-aware fine-tuning) shows ~0% inference
speedup versus the equivalent dense model when deployed with TensorRT-LLM, despite the
weights being verified as genuine, hardware-valid 2:4.

Verification of the sparse checkpoint:

  • GEMM weights: 50.00% zeros (dense baseline: 0.00%)
  • LayerNorm / embedding weights correctly untouched (0% zeros)

TensorRT-LLM engine is built after PTS+SAT -> TRTLLM ckpt export:

  1. --gemm_plugin auto --weight_sparsity — GEMMs routed to cuBLASLt (dense path).

Impact: should-have. The full sparsification cost (pruning + fine-tune + build)
is paid, but yields no throughput/latency benefit and no memory-footprint reduction on this
configuration.

Open question for maintainers: is any speedup expected for FP16 2:4 in a decode-bound /
small-batch regime on Ada (L4), and does the stock trtllm-build path actually
dispatch sparse (cuSPARSELt / native sparse) GEMM kernels for Qwen3 linear layers? If not,
what precision (FP8 / INT8), batch regime, and GPU architecture are required for ModelOpt 2:4
sparsity to show a measurable gain?

Steps/Code to reproduce bug

  1. Sparsify Qwen3-4B-Instruct-2507 to 2:4 with ModelOpt (SparseGPT + sparsity-aware
    fine-tuning) and export to a TensorRT-LLM checkpoint.(Used process in https://github.com/NVIDIA/Model-Optimizer/blob/main/examples/llm_sparsity/weight_sparsity)

  2. Verify the checkpoint is truly 2:4 (50% zeros in GEMM weights; every 4-group has exactly
    2 zeros on the last dim; norms untouched).

  3. Build the sparse engine:

    trtllm-build \
  --checkpoint_dir <sparse_ckpt> \
  --output_dir <sparse_engine> \
  --gemm_plugin auto \
  --weight_sparsity \
  --profiling_verbosity detailed \
  --max_batch_size 64 --max_input_len 4096 --max_seq_len 8192 --max_num_tokens 32768

  4. Build a dense engine identically, but without --weight_sparsity.

  5. Serve each engine (trtllm-serve, --backend tensorrt, --tp_size 1, port 8000).

  6. Benchmark both with guidellm, fixed shape prompt_tokens=512, output_tokens=256,
    --profile concurrent --rate 1,2,4,8,16,24,32,64,128 --max-requests 200 --max-seconds 120.

Result: sparse ≈ dense on all metrics; both plateau at ~485 gen tok/s around concurrency
~32; ITL identical (~33.8 ms). No speedup from 2:4.

Expected behavior

Either:

  • a measurable GEMM / throughput speedup from 2:4 structured sparsity on Sparse Tensor Cores, or
  • If it don't work on Ada Lovelace GPUS, Will it work on A10G GPU? or only A100
Who can help?

System information

  • Container used (if applicable): TensorRT-LLM container (build + serve); ModelOpt installed via pip
  • OS (e.g., Ubuntu 22.04): Ubuntu 24.04.4 LTS
  • CPU architecture (x86_64, aarch64): x86_64
  • GPU name (e.g. H100, A100, L40S): NVIDIA L4 (Ada Lovelace, sm_89, AWS g6.xlarge)
  • GPU memory size: 24 GB (23034 MiB)
  • Number of GPUs: 1
  • Library versions (if applicable):
    • Python: 3.12.3
    • ModelOpt version or commit hash: 0.45.0
    • CUDA: 13.2 (driver / nvidia-smi); PyTorch built against CUDA 13.1; NVIDIA driver 595.58.03
    • PyTorch: 2.11.0a0+eb65b36914.nv26.02
    • Transformers: 5.5.4
    • TensorRT-LLM: 1.3.0rc18
    • ONNXRuntime: not installed (N/A)
    • TensorRT: 10.15.1.29
  • Any other details that may help:
    • Model: Qwen3-4B-Instruct-2507, 2:4 structured-sparse, FP16.
    • Sparsity verified real (50% zeros; last-dim 2:4-valid = 1.0; norms untouched).
    • Two build variants tested (--gemm_plugin auto and --gemm_plugin disable --multiple_profiles enable); both give ~0% vs dense.
    • Footprint not reduced by 2:4 (non-zeros stored in FP16 + metadata); the --gemm_plugin disable engine is ~12 GB vs ~8.9 GB dense.
    • Workload is decode-bound / small-to-moderate batch, which may explain the lack of gain; seeking confirmation whether this is expected rather than a defect.

##Help
I already performed quantization techniques. And I want to know if performing sparsity along with quantization worth it? because I didn't see any performance speedup on sparsity alone.

Lingua principale
Python
Stelle
5.3k
Fork
745
Merge medio
4g 2h
PR unite (30g)
160

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/Model-Optimizer

Tutte le issue di NVIDIA/Model-Optimizer

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.