Wrong NVFP4 results of TensorRT 11.4 at batch > 1 when Q/K/V MatMuls share a TRT_FP4DynamicQuantize input, on RTX PRO 6000 Blackwell
Los mantenedores suelen responder en 6 días
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 8/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- python
- Área
- machine-learning
Línea de trabajo
The fix lives inside TensorRT's closed-source builder, so start by running python trt_fp4_batch_repro.py --level 3 and --level 0 to confirm the batch-2 error against the numpy reference. The reporter's layer information points at the __myl_ReshReshTran transpose of the FP4 activation feeding the value GEMM, while its block scales skip that reorder. Done would mean the level-3 batch-2 error matches the level-0 value of about 0.0195, which needs a TensorRT maintainer rather than a patch from a newcomer.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Description
An NVFP4 self-attention block exported the way ModelOpt does it gives each sequence a result that depends on the rest of the batch. Q, K and V are separate MatMuls whose shared input goes through TRT_FP4DynamicQuantize and the two-DequantizeLinear chain. Batch 1 matches a numpy reference, batch 2 does not, and builder optimization level 0 is correct (numbers below).
On a full encoder (BAAI/bge-reranker-v2-m3, XLM-R large), per-tensor dumps show that only the value projection is wrong: Q and K stay within 1.4% of fake quant while V is off by 44%. The engine's V is reproduced within 1.6% in PyTorch by pairing the FP4 values in (seq, batch) row order with the block scales in (batch, seq) order. In the layer information, a __myl_ReshReshTran kernel transposes the FP4 activation for the V GEMM, while its block scales go through the same swizzle kernel as for Q and K (__myl_MoveReplConcReshReshMove, same hash). The FP4 rows are reordered from (batch, seq) to (seq, batch) and their scales are not.
- Affected: TensorRT 10.16, 11.3.0.99 and 11.4.0.106 at builder optimization level 1 or higher.
- Not affected: optimization level 0; the same graph with Q/K/V as one MatMul; the GQA model of ModelOpt's
llama-nemotron-rerankexample.
On the encoder this is a silent ranking collapse at batch > 1: Recall@10 on Ko-StrategyQA (50 candidates per query) is 27.9%, against 87.3% for the same model exported with Q/K/V as one MatMul and 87.1% for FP8.
Environment
TensorRT Version: 11.4.0.106 (also 11.3.0.99 and 10.16)
NVIDIA GPU: RTX PRO 6000 Blackwell Server Edition (SM120)
NVIDIA Driver Version: 595.91.07
CUDA Version: 13.4
CUDNN Version: not used
Operating System: Ubuntu 24.04 (GCP g4-standard-48)
Python Version (if applicable): 3.12
PyTorch Version (if applicable): not needed for the repro
Baremetal or Container (if so, version): nvcr.io/nvidia/tritonserver:26.09-py3 (TensorRT 11.3.0.99); 11.4 and 10.16 from the tensorrt wheels in the same container
Relevant Files
Model link: none needed. The script below builds the attention block with random NVFP4 weights and checks it against numpy.
Steps To Reproduce
Commands or scripts: python trt_fp4_batch_repro.py --level 3 and --level 0 (needs tensorrt, onnx, numpy, polygraphy and an SM100/SM120 GPU).
Output on the RTX PRO 6000 Blackwell Server Edition (relative error of the engine output against the numpy reference, same inputs):
TensorRT 10.16.1.11, builder optimization level 0: relative error against numpy
each sequence alone (batch 1): 0.0195
both sequences together (batch 2): 0.0195
TensorRT 10.16.1.11, builder optimization level 3: relative error against numpy
each sequence alone (batch 1): 0.0196
both sequences together (batch 2): 1.2525
TensorRT 11.3.0.99, builder optimization level 0: relative error against numpy
each sequence alone (batch 1): 0.0195
both sequences together (batch 2): 0.0195
TensorRT 11.3.0.99, builder optimization level 3: relative error against numpy
each sequence alone (batch 1): 0.0196
both sequences together (batch 2): 1.2525
TensorRT 11.4.0.106, builder optimization level 0: relative error against numpy
each sequence alone (batch 1): 0.0195
both sequences together (batch 2): 0.0195
TensorRT 11.4.0.106, builder optimization level 3: relative error against numpy
each sequence alone (batch 1): 0.0196
both sequences together (batch 2): 1.2525
About 0.02 is what a correct engine reaches against this numpy reference (level 0, and each sequence alone at level 3); at level 3 the batched result is off by 125%.
trt_fp4_batch_repro.py
"""Standalone TensorRT repro: an NVFP4 attention block whose result for one sequence changes with the rest of the batch.
The graph is one self-attention block as ModelOpt exports it: the input goes through TRT_FP4DynamicQuantize and two
DequantizeLinear nodes (FP8 block scales over 16 elements, then FP4 data) into the Q, K and V MatMuls, whose NVFP4
weights are initializers dequantized the same way; then the usual reshape and transposes, softmax(Q K^T / 8) V, and
the heads merged back. Random weights, no model download.
Each sequence of a batch is independent, so running a batch of 2 must give each sequence the result it gets alone.
On TensorRT 10.16 to 11.4 at builder optimization level >= 1 it does not: the engine feeds the value GEMM the FP4 input
with its rows reordered from (batch, seq) to (seq, batch) but keeps the block scales in (batch, seq) order.
usage: python trt_fp4_batch_repro.py [--level N] [--save model.onnx]
needs: tensorrt, onnx, numpy, polygraphy, an SM100/SM120 GPU
"""
from __future__ import annotations
import argparse
import numpy as np
import onnx
import tensorrt as trt
from onnx import TensorProto, helper, numpy_helper
from polygraphy.backend.trt import CreateConfig, EngineFromNetwork, NetworkFromOnnxBytes, Profile, TrtRunner
HIDDEN, HEADS, HEAD_DIM, BLOCK = 1024, 16, 64, 16
X_AMAX = 16.0
E2M1 = np.array([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0], dtype=np.float32)
def e4m3_values() -> np.ndarray:
"""Every finite non-negative FP8 E4M3 value, indexed by its bit pattern."""
bits = np.arange(127)
exponent, mantissa = bits >> 3, (bits & 7).astype(np.float32)
return np.where(exponent == 0, mantissa / 8 * 2.0 ** -6, (1 + mantissa / 8) * 2.0 ** (exponent - 7.0))
E4M3 = e4m3_values()
def nearest(values: np.ndarray, grid: np.ndarray) -> np.ndarray:
return grid[np.abs(values[..., None] - grid).argmin(axis=-1)]
def nvfp4_weight(name: str, rng: np.random.Generator) -> tuple[list[onnx.TensorProto], list[onnx.NodeProto]]:
"""A [HIDDEN, HIDDEN] NVFP4 weight: random E2M1 codes, power-of-two FP8 block scales (exact in E4M3), and a
per-tensor scale; dequantized by DQ(block scales, per-tensor scale) -> DQ(data, scales, axis=-1, block_size=16)."""
codes = rng.integers(0, 16, size=(HIDDEN, HIDDEN), dtype=np.uint8) # bit 3 is the sign
packed = (codes[:, 0::2] | (codes[:, 1::2] << 4)).astype(np.uint8)
exponents = rng.integers(-2, 3, size=(HIDDEN, HIDDEN // BLOCK))
scale_bytes = ((exponents + 7) << 3).astype(np.uint8) # E4M3 bits of 2**exponent
bias = rng.normal(0, 0.1, HIDDEN).astype(np.float16)
values = np.where(codes & 8, -1.0, 1.0) * E2M1[codes & 7]
WEIGHTS[name] = (values * np.repeat(2.0 ** exponents, BLOCK, axis=1) * 0.002, bias.astype(np.float32))
initializers = [
helper.make_tensor(f"{name}_f4", TensorProto.FLOAT4E2M1, [HIDDEN, HIDDEN], packed.tobytes(), raw=True),
helper.make_tensor(f"{name}_sf", TensorProto.FLOAT8E4M3FN, [HIDDEN, HIDDEN // BLOCK], scale_bytes.tobytes(),
raw=True),
numpy_helper.from_array(np.array(0.002, dtype=np.float16), f"{name}_gs"),
numpy_helper.from_array(bias, f"{name}_bias"),
]
nodes = [
helper.make_node("DequantizeLinear", [f"{name}_sf", f"{name}_gs"], [f"{name}_scales"]),
helper.make_node("DequantizeLinear", [f"{name}_f4", f"{name}_scales"], [f"{name}_w"], axis=-1,
block_size=BLOCK),
helper.make_node("Transpose", [f"{name}_w"], [f"{name}_wt"], perm=[1, 0]),
]
return initializers, nodes
WEIGHTS: dict[str, tuple[np.ndarray, np.ndarray]] = {} # name -> (dequantized weight [out, in], bias)
def build_graph(rng: np.random.Generator) -> onnx.ModelProto:
initializers = [numpy_helper.from_array(np.array(X_AMAX / 2688.0, dtype=np.float16), "x_gs"),
numpy_helper.from_array(np.array([0, 0, HEADS, HEAD_DIM], dtype=np.int64), "split_heads"),
numpy_helper.from_array(np.array([0, 0, HIDDEN], dtype=np.int64), "merge_heads"),
numpy_helper.from_array(np.array(HEAD_DIM ** -0.5, dtype=np.float16), "softmax_scale")]
nodes = []
for name in ("q", "k", "v"):
weight_initializers, weight_nodes = nvfp4_weight(name, rng)
initializers += weight_initializers
nodes += weight_nodes
nodes += [
helper.make_node("TRT_FP4DynamicQuantize", ["x", "x_gs"], [f"{name}_x_f4", f"{name}_x_sf"], domain="trt",
axis=-1, block_size=BLOCK, scale_type=TensorProto.FLOAT8E4M3FN),
helper.make_node("DequantizeLinear", [f"{name}_x_sf", "x_gs"], [f"{name}_x_scales"]),
helper.make_node("DequantizeLinear", [f"{name}_x_f4", f"{name}_x_scales"], [f"{name}_xq"], axis=-1,
block_size=BLOCK),
helper.make_node("MatMul", [f"{name}_xq", f"{name}_wt"], [f"{name}_mm"]),
helper.make_node("Add", [f"{name}_mm", f"{name}_bias"], [f"{name}_out"]),
helper.make_node("Reshape", [f"{name}_out", "split_heads"], [f"{name}_heads"]),
]
nodes += [
helper.make_node("Transpose", ["q_heads"], ["q_t"], perm=[0, 2, 1, 3]),
helper.make_node("Transpose", ["k_heads"], ["k_t"], perm=[0, 2, 3, 1]),
helper.make_node("Transpose", ["v_heads"], ["v_t"], perm=[0, 2, 1, 3]),
helper.make_node("Mul", ["q_t", "softmax_scale"], ["q_scaled"]),
helper.make_node("MatMul", ["q_scaled", "k_t"], ["scores"]),
helper.make_node("Softmax", ["scores"], ["probs"], axis=-1),
helper.make_node("MatMul", ["probs", "v_t"], ["context"]),
helper.make_node("Transpose", ["context"], ["context_t"], perm=[0, 2, 1, 3]),
helper.make_node("Reshape", ["context_t", "merge_heads"], ["y"]),
]
graph = helper.make_graph(
nodes, "nvfp4_attention",
[helper.make_tensor_value_info("x", TensorProto.FLOAT16, ["batch", "seq", HIDDEN])],
[helper.make_tensor_value_info("y", TensorProto.FLOAT16, ["batch", "seq", HIDDEN])],
initializers)
return helper.make_model(graph, opset_imports=[helper.make_opsetid("", 23), helper.make_opsetid("trt", 1)])
def reference(x: np.ndarray) -> np.ndarray:
"""The graph in numpy: dynamic NVFP4 of x (E4M3 block scales of amax / 6 over the per-tensor scale, E2M1 values),
the three projections, and attention per sequence."""
global_scale = np.float32(np.float16(X_AMAX / 2688.0))
blocks = x.astype(np.float32).reshape(*x.shape[:-1], -1, BLOCK)
scales = nearest(np.abs(blocks).max(axis=-1, keepdims=True) / 6 / global_scale, E4M3) * global_scale
scaled = blocks / np.where(scales == 0, 1, scales)
xq = (np.sign(scaled) * nearest(np.minimum(np.abs(scaled), 6), E2M1) * scales).reshape(x.shape)
heads = {}
for name, (weight, bias) in WEIGHTS.items():
heads[name] = (xq @ weight.T + bias).reshape(*x.shape[:-1], HEADS, HEAD_DIM).transpose(0, 2, 1, 3)
scores = heads["q"] @ heads["k"].transpose(0, 1, 3, 2) * HEAD_DIM ** -0.5
probs = np.exp(scores - scores.max(axis=-1, keepdims=True))
probs /= probs.sum(axis=-1, keepdims=True)
return (probs @ heads["v"]).transpose(0, 2, 1, 3).reshape(x.shape)
def relative(a: np.ndarray, b: np.ndarray) -> float:
return float(np.linalg.norm(a - b) / np.linalg.norm(b))
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--level", type=int, default=3, help="builder optimization level")
parser.add_argument("--save", help="also write the ONNX graph here")
args = parser.parse_args()
rng = np.random.default_rng(0)
model = build_graph(rng)
if args.save:
onnx.save(model, args.save)
profile = Profile().add("x", min=(1, 16, HIDDEN), opt=(4, 128, HIDDEN), max=(8, 256, HIDDEN))
network = NetworkFromOnnxBytes(model.SerializeToString(), strongly_typed=True)
engine = EngineFromNetwork(network, config=CreateConfig(profiles=[profile],
builder_optimization_level=args.level))
# Tokens of different sizes, as in real activations, so that one token's block scales misfit another's values.
token_size = rng.lognormal(0, 0.7, size=(2, 128, 1))
x = np.clip(rng.normal(0, 1, size=(2, 128, HIDDEN)) * token_size, -X_AMAX, X_AMAX).astype(np.float16)
with TrtRunner(engine) as runner:
together = runner.infer({"x": x})["y"].astype(np.float32)
alone = np.stack([runner.infer({"x": x[row:row + 1]})["y"].astype(np.float32)[0] for row in range(2)])
expected = reference(x)
print(f"TensorRT {trt.__version__}, builder optimization level {args.level}: relative error against numpy")
print(f" each sequence alone (batch 1): {relative(alone, expected):.4f}")
print(f" both sequences together (batch 2): {relative(together, expected):.4f}")
if __name__ == "__main__":
main()
Have you tried the latest release?: Yes, 11.4.0.106 is affected.
Can this model run on other frameworks? ONNX Runtime does not implement TRT_FP4DynamicQuantize. The script's numpy reference implements the same graph: dynamic NVFP4 of the input, the three projections, and attention per sequence.
- Lenguaje dominante
- C++
- Estrellas
- 13.5k
- Forks
- 2.4k
- Merge medio
- 6 d 21 h
- PR fusionados (30 d)
- 3
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de NVIDIA/TensorRT
-
Module:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
Los mantenedores suelen responder en 6 días
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyPosiblemente ocupada @0z5a la tomó hace 16 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 6 días
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerancePosiblemente ocupada @0z5a la tomó hace 16 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 6 días
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodePosiblemente ocupada @gabortar la tomó hace 39 días. AbiertoModule:GraphSurgeon
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 6 días
-
Polygraphy parse_meta accepts a shape with a missing bracketPosiblemente ocupada @SashaMIT la tomó hace 51 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
Los mantenedores suelen responder en 6 días
Todos los issues de NVIDIA/TensorRT
Issues similares
-
[UBSan] spawn_simulator passes null child_options.args to uv_spawn (execvp nonnull violation)Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 79/100
tenstorrent/tt-umd#4017 ·
Los mantenedores suelen responder en 3 días
-
difficulty:easy func:gui prio:low type:bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
NatronGitHub/Natron#1538 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
hyprwm/hyprsunset#103 · 1 comentario ·
-
Feature request
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
qbittorrent/qBittorrent#25338 ·
Los mantenedores suelen responder en 2 días
-
Needs: Attention Needs: Version Info Platform: Android Platform: tvOS
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
react/react-native#60180 · 3 comentarios ·
Los mantenedores suelen responder en 2 días