[Bug]: Inference speed becomes slower after AutoRound quantization

The payload points to the Qwen3 MoE path: the log shows transformers' Qwen3MoeSparseMoeBlock being patched to auto_round.modeling.unfused_moe.qwen3_moe.LinearQwen3MoeSparseMoeBlock, and the export is the auto_round format loaded through from_pretrained. Start by profiling generation for the original and quantized models to find where the time goes, then check the unfused MoE block and the W4A16 kernel selection on load. Done means a documented cause and a measured speedup, not just the reported 5x gap.

Problem Description

I observed that the inference speed of models quantized with AutoRound is slower than the original model in Qwen3-30B-A2B.

Reproduction Steps

1. autoround quant

from auto_round import AutoRound
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name_or_path = "/data8/models/Qwen/Qwen3-30B-A3B"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path)

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
# 可选量化配置：
# "W2A16", "W3A16", "W4A16", "W8A16", "NVFP4", "MXFP4"（无真实 kernel）, "GGUF:Q4_K_M" 等
ar = AutoRound(model, tokenizer, scheme="W4A16", nsamples=128, iters=50, lr=5e-3)

# 追求高模型精度（速度会慢 4–5 倍）
# `low_gpu_mem_usage=True` 可节省 ~20GB 显存，但会慢 ~30%
# ar = AutoRound(model_name_or_path, nsamples=512, iters=1000, low_gpu_mem_usage=True)

# 追求量化速度（提速 2–3 倍），但在 W4G128 下精度会略微下降
# ar = AutoRound(model_name_or_path, nsamples=128, iters=50, lr=5e-3)

# 支持的导出格式："auto_round"（默认）, "auto_gptq", "auto_awq", "llm_compressor", "gguf:q4_k_m" 等
ar.quantize_and_save(output_dir="./qmodel", format="auto_round")

2. inference

import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# model_name = "/data8/models/Qwen/Qwen3-30B-A3B"
model_name = "/data8/Quant/qmodel/Qwen3-30B-A3B-w4g128"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="cuda",
    torch_dtype="auto",
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True,
)

text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)

print("warmup...")
with torch.no_grad():
    _ = model.generate(
        **inputs,
        max_new_tokens=20,
        do_sample=False,
    )
torch.cuda.synchronize()

num_tokens = 120
start = time.time()

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=num_tokens,
        do_sample=False,
    )
torch.cuda.synchronize()

end = time.time()
elapsed = end - start
generated_tokens = output.shape[-1] - inputs.input_ids.shape[-1]
print(tokenizer.decode(output[0]))
print("====================")
print(f"time: {elapsed:.3f}s")
print(f"tokens: {generated_tokens}")
print(f"tok/s: {generated_tokens / elapsed:.2f}")

Environment Information

• Python version :3.10.21
• GPT-QModel : 7.0.0
• Transformers : 5.9.0
• Torch : 2.11.0+cu126
• Triton : 3.6.0

Error Logs

Failed to load /data3/zhangdeming/miniconda3/envs/llmcompressor/lib/python3.10/site-packages/torchao/_C_cutlass_90a.abi3.so: Could not load this library: /data3/zhangdeming/miniconda3/envs/llmcompressor/lib/python3.10/site-packages/torchao/_C_cutlass_90a.abi3.so
Failed to load /data3/zhangdeming/miniconda3/envs/llmcompressor/lib/python3.10/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so: Could not load this library: /data3/zhangdeming/miniconda3/envs/llmcompressor/lib/python3.10/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 531/531 [00:11<00:00, 47.97it/s]
warmup...
There is a girl who likes adventure, and she is going to a place where there are 1000 islands. She is on the first island, and wants to get to the 1000th island. The only way to move is by boat. Each boat has a certain number of seats, and each seat can hold one person. The boat can only go from one island to another if there is a seat available. There are 1000 boats, each with a different number of seats, from 1 to 1000. How can she get to the 1000th
====================
time: 5.112s
tokens: 120
tok/s: 23.47


Failed to load /data3/zhangdeming/miniconda3/envs/llmcompressor/lib/python3.10/site-packages/torchao/_C_cutlass_90a.abi3.so: Could not load this library: /data3/zhangdeming/miniconda3/envs/llmcompressor/lib/python3.10/site-packages/torchao/_C_cutlass_90a.abi3.so
Failed to load /data3/zhangdeming/miniconda3/envs/llmcompressor/lib/python3.10/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so: Could not load this library: /data3/zhangdeming/miniconda3/envs/llmcompressor/lib/python3.10/site-packages/torchao/_C_mxfp8.cpython-310-x86_64-linux-gnu.so
2026-10-13 11:10:50 INFO __init__.py L246: Patched module: transformers.models.qwen3_moe.modeling_qwen3_moe.Qwen3MoeSparseMoeBlock -> auto_round.modeling.unfused_moe.qwen3_moe.LinearQwen3MoeSparseMoeBlock

INFO  LogBar: headless/CI mode; progress animations disabled.                                                                                               
WARN  Python GIL is enabled: Multi-gpu quant acceleration for MoE models is sub-optimal and multi-core accelerated cpu packing is also disabled. We recommend Python >= 3.13.3t with Pytorch > 2.8 for mult-gpu quantization and multi-cpu packing with env `PYTHON_GIL=0`.
INFO  ENV: Auto setting PYTORCH_ALLOC_CONF='expandable_segments:True,max_split_size_mb:256,garbage_collection_threshold:0.7' for memory saving.             
INFO  ENV: Auto setting CUDA_DEVICE_ORDER=PCI_BUS_ID for correctness.                                                                                       
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
INFO  

┌─────────────┐    ┌────────────────────────┐    ┌────────────┐    ┌─────────┐
│ GPT-QModel  │ -> │ ▓▓▓▓▓▓▓▓▓▓▓▓ 16bit     │ -> │ ▒▒▒▒ 8bit  │ -> │ ░░ 4bit │
└─────────────┘    └────────────────────────┘    └────────────┘    └─────────┘
GPT-QModel   : 7.0.0
Transformers : 5.9.0
Torch        : 2.11.0+cu126
Triton       : 3.6.0
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 56115/56115 [00:09<00:00, 5659.16it/s]
INFO  gc.collect() reclaimed 0 objects in 0.762s                                                                                                            
warmup...
There is a girl who likes adventure, and she is going to go to the forest to find a rare flower. She has a map that shows the forest as a grid. The grid has 5 rows and 5 columns. Each cell in the grid can be either a tree, a flower, or a path. The girl starts at the top-left corner (0,0) and needs to reach thebottom-right corner (4,4). The girl can move in four directions: up, down, left, right. However, she cannot step on a tree. The flower is located at (2,3).The girl wants
====================
time: 33.782s
tokens: 120
tok/s: 3.55

Additional Context

The inference speed of the quantized model is more than 5 times slower than the original model (5.112s → 33.78s).