Wrong NVFP4 results of TensorRT 11.4 at batch > 1 when Q/K/V MatMuls share a TRT_FP4DynamicQuantize input, on RTX PRO 6000 Blackwell

未关闭
#4,879 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 6 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
8/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python

调研方向

The fix lives inside TensorRT's closed-source builder, so start by running python trt_fp4_batch_repro.py --level 3 and --level 0 to confirm the batch-2 error against the numpy reference. The reporter's layer information points at the __myl_ReshReshTran transpose of the FP4 activation feeding the value GEMM, while its block scales skip that reorder. Done would mean the level-3 batch-2 error matches the level-0 value of about 0.0195, which needs a TensorRT maintainer rather than a patch from a newcomer.

由索引模型根据 Issue 内容生成。

描述

Module:Accuracy

Description

An NVFP4 self-attention block exported the way ModelOpt does it gives each sequence a result that depends on the rest of the batch. Q, K and V are separate MatMuls whose shared input goes through TRT_FP4DynamicQuantize and the two-DequantizeLinear chain. Batch 1 matches a numpy reference, batch 2 does not, and builder optimization level 0 is correct (numbers below).

On a full encoder (BAAI/bge-reranker-v2-m3, XLM-R large), per-tensor dumps show that only the value projection is wrong: Q and K stay within 1.4% of fake quant while V is off by 44%. The engine's V is reproduced within 1.6% in PyTorch by pairing the FP4 values in (seq, batch) row order with the block scales in (batch, seq) order. In the layer information, a __myl_ReshReshTran kernel transposes the FP4 activation for the V GEMM, while its block scales go through the same swizzle kernel as for Q and K (__myl_MoveReplConcReshReshMove, same hash). The FP4 rows are reordered from (batch, seq) to (seq, batch) and their scales are not.

  • Affected: TensorRT 10.16, 11.3.0.99 and 11.4.0.106 at builder optimization level 1 or higher.
  • Not affected: optimization level 0; the same graph with Q/K/V as one MatMul; the GQA model of ModelOpt's llama-nemotron-rerank example.

On the encoder this is a silent ranking collapse at batch > 1: Recall@10 on Ko-StrategyQA (50 candidates per query) is 27.9%, against 87.3% for the same model exported with Q/K/V as one MatMul and 87.1% for FP8.

Environment

TensorRT Version: 11.4.0.106 (also 11.3.0.99 and 10.16)

NVIDIA GPU: RTX PRO 6000 Blackwell Server Edition (SM120)

NVIDIA Driver Version: 595.91.07

CUDA Version: 13.4

CUDNN Version: not used

Operating System: Ubuntu 24.04 (GCP g4-standard-48)

Python Version (if applicable): 3.12

PyTorch Version (if applicable): not needed for the repro

Baremetal or Container (if so, version): nvcr.io/nvidia/tritonserver:26.09-py3 (TensorRT 11.3.0.99); 11.4 and 10.16 from the tensorrt wheels in the same container

Relevant Files

Model link: none needed. The script below builds the attention block with random NVFP4 weights and checks it against numpy.

Steps To Reproduce

Commands or scripts: python trt_fp4_batch_repro.py --level 3 and --level 0 (needs tensorrt, onnx, numpy, polygraphy and an SM100/SM120 GPU).

Output on the RTX PRO 6000 Blackwell Server Edition (relative error of the engine output against the numpy reference, same inputs):

TensorRT 10.16.1.11, builder optimization level 0: relative error against numpy
  each sequence alone (batch 1): 0.0195
  both sequences together (batch 2): 0.0195
TensorRT 10.16.1.11, builder optimization level 3: relative error against numpy
  each sequence alone (batch 1): 0.0196
  both sequences together (batch 2): 1.2525
TensorRT 11.3.0.99, builder optimization level 0: relative error against numpy
  each sequence alone (batch 1): 0.0195
  both sequences together (batch 2): 0.0195
TensorRT 11.3.0.99, builder optimization level 3: relative error against numpy
  each sequence alone (batch 1): 0.0196
  both sequences together (batch 2): 1.2525
TensorRT 11.4.0.106, builder optimization level 0: relative error against numpy
  each sequence alone (batch 1): 0.0195
  both sequences together (batch 2): 0.0195
TensorRT 11.4.0.106, builder optimization level 3: relative error against numpy
  each sequence alone (batch 1): 0.0196
  both sequences together (batch 2): 1.2525

About 0.02 is what a correct engine reaches against this numpy reference (level 0, and each sequence alone at level 3); at level 3 the batched result is off by 125%.

trt_fp4_batch_repro.py
"""Standalone TensorRT repro: an NVFP4 attention block whose result for one sequence changes with the rest of the batch.

The graph is one self-attention block as ModelOpt exports it: the input goes through TRT_FP4DynamicQuantize and two
DequantizeLinear nodes (FP8 block scales over 16 elements, then FP4 data) into the Q, K and V MatMuls, whose NVFP4
weights are initializers dequantized the same way; then the usual reshape and transposes, softmax(Q K^T / 8) V, and
the heads merged back. Random weights, no model download.

Each sequence of a batch is independent, so running a batch of 2 must give each sequence the result it gets alone.
On TensorRT 10.16 to 11.4 at builder optimization level >= 1 it does not: the engine feeds the value GEMM the FP4 input
with its rows reordered from (batch, seq) to (seq, batch) but keeps the block scales in (batch, seq) order.
usage: python trt_fp4_batch_repro.py [--level N] [--save model.onnx]
needs: tensorrt, onnx, numpy, polygraphy, an SM100/SM120 GPU
"""
from __future__ import annotations

import argparse

import numpy as np
import onnx
import tensorrt as trt
from onnx import TensorProto, helper, numpy_helper
from polygraphy.backend.trt import CreateConfig, EngineFromNetwork, NetworkFromOnnxBytes, Profile, TrtRunner

HIDDEN, HEADS, HEAD_DIM, BLOCK = 1024, 16, 64, 16
X_AMAX = 16.0
E2M1 = np.array([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0], dtype=np.float32)


def e4m3_values() -> np.ndarray:
    """Every finite non-negative FP8 E4M3 value, indexed by its bit pattern."""
    bits = np.arange(127)
    exponent, mantissa = bits >> 3, (bits & 7).astype(np.float32)
    return np.where(exponent == 0, mantissa / 8 * 2.0 ** -6, (1 + mantissa / 8) * 2.0 ** (exponent - 7.0))


E4M3 = e4m3_values()


def nearest(values: np.ndarray, grid: np.ndarray) -> np.ndarray:
    return grid[np.abs(values[..., None] - grid).argmin(axis=-1)]


def nvfp4_weight(name: str, rng: np.random.Generator) -> tuple[list[onnx.TensorProto], list[onnx.NodeProto]]:
    """A [HIDDEN, HIDDEN] NVFP4 weight: random E2M1 codes, power-of-two FP8 block scales (exact in E4M3), and a
    per-tensor scale; dequantized by DQ(block scales, per-tensor scale) -> DQ(data, scales, axis=-1, block_size=16)."""
    codes = rng.integers(0, 16, size=(HIDDEN, HIDDEN), dtype=np.uint8)  # bit 3 is the sign
    packed = (codes[:, 0::2] | (codes[:, 1::2] << 4)).astype(np.uint8)
    exponents = rng.integers(-2, 3, size=(HIDDEN, HIDDEN // BLOCK))
    scale_bytes = ((exponents + 7) << 3).astype(np.uint8)  # E4M3 bits of 2**exponent
    bias = rng.normal(0, 0.1, HIDDEN).astype(np.float16)
    values = np.where(codes & 8, -1.0, 1.0) * E2M1[codes & 7]
    WEIGHTS[name] = (values * np.repeat(2.0 ** exponents, BLOCK, axis=1) * 0.002, bias.astype(np.float32))
    initializers = [
        helper.make_tensor(f"{name}_f4", TensorProto.FLOAT4E2M1, [HIDDEN, HIDDEN], packed.tobytes(), raw=True),
        helper.make_tensor(f"{name}_sf", TensorProto.FLOAT8E4M3FN, [HIDDEN, HIDDEN // BLOCK], scale_bytes.tobytes(),
                           raw=True),
        numpy_helper.from_array(np.array(0.002, dtype=np.float16), f"{name}_gs"),
        numpy_helper.from_array(bias, f"{name}_bias"),
    ]
    nodes = [
        helper.make_node("DequantizeLinear", [f"{name}_sf", f"{name}_gs"], [f"{name}_scales"]),
        helper.make_node("DequantizeLinear", [f"{name}_f4", f"{name}_scales"], [f"{name}_w"], axis=-1,
                         block_size=BLOCK),
        helper.make_node("Transpose", [f"{name}_w"], [f"{name}_wt"], perm=[1, 0]),
    ]
    return initializers, nodes


WEIGHTS: dict[str, tuple[np.ndarray, np.ndarray]] = {}  # name -> (dequantized weight [out, in], bias)


def build_graph(rng: np.random.Generator) -> onnx.ModelProto:
    initializers = [numpy_helper.from_array(np.array(X_AMAX / 2688.0, dtype=np.float16), "x_gs"),
                    numpy_helper.from_array(np.array([0, 0, HEADS, HEAD_DIM], dtype=np.int64), "split_heads"),
                    numpy_helper.from_array(np.array([0, 0, HIDDEN], dtype=np.int64), "merge_heads"),
                    numpy_helper.from_array(np.array(HEAD_DIM ** -0.5, dtype=np.float16), "softmax_scale")]
    nodes = []
    for name in ("q", "k", "v"):
        weight_initializers, weight_nodes = nvfp4_weight(name, rng)
        initializers += weight_initializers
        nodes += weight_nodes
        nodes += [
            helper.make_node("TRT_FP4DynamicQuantize", ["x", "x_gs"], [f"{name}_x_f4", f"{name}_x_sf"], domain="trt",
                             axis=-1, block_size=BLOCK, scale_type=TensorProto.FLOAT8E4M3FN),
            helper.make_node("DequantizeLinear", [f"{name}_x_sf", "x_gs"], [f"{name}_x_scales"]),
            helper.make_node("DequantizeLinear", [f"{name}_x_f4", f"{name}_x_scales"], [f"{name}_xq"], axis=-1,
                             block_size=BLOCK),
            helper.make_node("MatMul", [f"{name}_xq", f"{name}_wt"], [f"{name}_mm"]),
            helper.make_node("Add", [f"{name}_mm", f"{name}_bias"], [f"{name}_out"]),
            helper.make_node("Reshape", [f"{name}_out", "split_heads"], [f"{name}_heads"]),
        ]
    nodes += [
        helper.make_node("Transpose", ["q_heads"], ["q_t"], perm=[0, 2, 1, 3]),
        helper.make_node("Transpose", ["k_heads"], ["k_t"], perm=[0, 2, 3, 1]),
        helper.make_node("Transpose", ["v_heads"], ["v_t"], perm=[0, 2, 1, 3]),
        helper.make_node("Mul", ["q_t", "softmax_scale"], ["q_scaled"]),
        helper.make_node("MatMul", ["q_scaled", "k_t"], ["scores"]),
        helper.make_node("Softmax", ["scores"], ["probs"], axis=-1),
        helper.make_node("MatMul", ["probs", "v_t"], ["context"]),
        helper.make_node("Transpose", ["context"], ["context_t"], perm=[0, 2, 1, 3]),
        helper.make_node("Reshape", ["context_t", "merge_heads"], ["y"]),
    ]
    graph = helper.make_graph(
        nodes, "nvfp4_attention",
        [helper.make_tensor_value_info("x", TensorProto.FLOAT16, ["batch", "seq", HIDDEN])],
        [helper.make_tensor_value_info("y", TensorProto.FLOAT16, ["batch", "seq", HIDDEN])],
        initializers)
    return helper.make_model(graph, opset_imports=[helper.make_opsetid("", 23), helper.make_opsetid("trt", 1)])


def reference(x: np.ndarray) -> np.ndarray:
    """The graph in numpy: dynamic NVFP4 of x (E4M3 block scales of amax / 6 over the per-tensor scale, E2M1 values),
    the three projections, and attention per sequence."""
    global_scale = np.float32(np.float16(X_AMAX / 2688.0))
    blocks = x.astype(np.float32).reshape(*x.shape[:-1], -1, BLOCK)
    scales = nearest(np.abs(blocks).max(axis=-1, keepdims=True) / 6 / global_scale, E4M3) * global_scale
    scaled = blocks / np.where(scales == 0, 1, scales)
    xq = (np.sign(scaled) * nearest(np.minimum(np.abs(scaled), 6), E2M1) * scales).reshape(x.shape)
    heads = {}
    for name, (weight, bias) in WEIGHTS.items():
        heads[name] = (xq @ weight.T + bias).reshape(*x.shape[:-1], HEADS, HEAD_DIM).transpose(0, 2, 1, 3)
    scores = heads["q"] @ heads["k"].transpose(0, 1, 3, 2) * HEAD_DIM ** -0.5
    probs = np.exp(scores - scores.max(axis=-1, keepdims=True))
    probs /= probs.sum(axis=-1, keepdims=True)
    return (probs @ heads["v"]).transpose(0, 2, 1, 3).reshape(x.shape)


def relative(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.linalg.norm(a - b) / np.linalg.norm(b))


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--level", type=int, default=3, help="builder optimization level")
    parser.add_argument("--save", help="also write the ONNX graph here")
    args = parser.parse_args()

    rng = np.random.default_rng(0)
    model = build_graph(rng)
    if args.save:
        onnx.save(model, args.save)
    profile = Profile().add("x", min=(1, 16, HIDDEN), opt=(4, 128, HIDDEN), max=(8, 256, HIDDEN))
    network = NetworkFromOnnxBytes(model.SerializeToString(), strongly_typed=True)
    engine = EngineFromNetwork(network, config=CreateConfig(profiles=[profile],
                                                            builder_optimization_level=args.level))
    # Tokens of different sizes, as in real activations, so that one token's block scales misfit another's values.
    token_size = rng.lognormal(0, 0.7, size=(2, 128, 1))
    x = np.clip(rng.normal(0, 1, size=(2, 128, HIDDEN)) * token_size, -X_AMAX, X_AMAX).astype(np.float16)
    with TrtRunner(engine) as runner:
        together = runner.infer({"x": x})["y"].astype(np.float32)
        alone = np.stack([runner.infer({"x": x[row:row + 1]})["y"].astype(np.float32)[0] for row in range(2)])
    expected = reference(x)
    print(f"TensorRT {trt.__version__}, builder optimization level {args.level}: relative error against numpy")
    print(f"  each sequence alone (batch 1): {relative(alone, expected):.4f}")
    print(f"  both sequences together (batch 2): {relative(together, expected):.4f}")


if __name__ == "__main__":
    main()

Have you tried the latest release?: Yes, 11.4.0.106 is affected.

Can this model run on other frameworks? ONNX Runtime does not implement TRT_FP4DynamicQuantize. The script's numpy reference implements the same graph: dynamic NVFP4 of the input, the three projections, and attention per sequence.

主要语言
C++
星标
13.5k
派生
2.4k
平均合并
6 天 21 小时
30 天内合并 PR
3

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

NVIDIA/TensorRT 的其他 Issue

查看 NVIDIA/TensorRT 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。