Wrong NVFP4 results of TensorRT 11.4 at batch > 1 when Q/K/V MatMuls share a TRT_FP4DynamicQuantize input, on RTX PRO 6000 Blackwell

Ouverte
#4,879 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 6 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
8/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python

Piste de recherche

The fix lives inside TensorRT's closed-source builder, so start by running python trt_fp4_batch_repro.py --level 3 and --level 0 to confirm the batch-2 error against the numpy reference. The reporter's layer information points at the __myl_ReshReshTran transpose of the FP4 activation feeding the value GEMM, while its block scales skip that reorder. Done would mean the level-3 batch-2 error matches the level-0 value of about 0.0195, which needs a TensorRT maintainer rather than a patch from a newcomer.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Module:Accuracy

Description

An NVFP4 self-attention block exported the way ModelOpt does it gives each sequence a result that depends on the rest of the batch. Q, K and V are separate MatMuls whose shared input goes through TRT_FP4DynamicQuantize and the two-DequantizeLinear chain. Batch 1 matches a numpy reference, batch 2 does not, and builder optimization level 0 is correct (numbers below).

On a full encoder (BAAI/bge-reranker-v2-m3, XLM-R large), per-tensor dumps show that only the value projection is wrong: Q and K stay within 1.4% of fake quant while V is off by 44%. The engine's V is reproduced within 1.6% in PyTorch by pairing the FP4 values in (seq, batch) row order with the block scales in (batch, seq) order. In the layer information, a __myl_ReshReshTran kernel transposes the FP4 activation for the V GEMM, while its block scales go through the same swizzle kernel as for Q and K (__myl_MoveReplConcReshReshMove, same hash). The FP4 rows are reordered from (batch, seq) to (seq, batch) and their scales are not.

  • Affected: TensorRT 10.16, 11.3.0.99 and 11.4.0.106 at builder optimization level 1 or higher.
  • Not affected: optimization level 0; the same graph with Q/K/V as one MatMul; the GQA model of ModelOpt's llama-nemotron-rerank example.

On the encoder this is a silent ranking collapse at batch > 1: Recall@10 on Ko-StrategyQA (50 candidates per query) is 27.9%, against 87.3% for the same model exported with Q/K/V as one MatMul and 87.1% for FP8.

Environment

TensorRT Version: 11.4.0.106 (also 11.3.0.99 and 10.16)

NVIDIA GPU: RTX PRO 6000 Blackwell Server Edition (SM120)

NVIDIA Driver Version: 595.91.07

CUDA Version: 13.4

CUDNN Version: not used

Operating System: Ubuntu 24.04 (GCP g4-standard-48)

Python Version (if applicable): 3.12

PyTorch Version (if applicable): not needed for the repro

Baremetal or Container (if so, version): nvcr.io/nvidia/tritonserver:26.09-py3 (TensorRT 11.3.0.99); 11.4 and 10.16 from the tensorrt wheels in the same container

Relevant Files

Model link: none needed. The script below builds the attention block with random NVFP4 weights and checks it against numpy.

Steps To Reproduce

Commands or scripts: python trt_fp4_batch_repro.py --level 3 and --level 0 (needs tensorrt, onnx, numpy, polygraphy and an SM100/SM120 GPU).

Output on the RTX PRO 6000 Blackwell Server Edition (relative error of the engine output against the numpy reference, same inputs):

TensorRT 10.16.1.11, builder optimization level 0: relative error against numpy
  each sequence alone (batch 1): 0.0195
  both sequences together (batch 2): 0.0195
TensorRT 10.16.1.11, builder optimization level 3: relative error against numpy
  each sequence alone (batch 1): 0.0196
  both sequences together (batch 2): 1.2525
TensorRT 11.3.0.99, builder optimization level 0: relative error against numpy
  each sequence alone (batch 1): 0.0195
  both sequences together (batch 2): 0.0195
TensorRT 11.3.0.99, builder optimization level 3: relative error against numpy
  each sequence alone (batch 1): 0.0196
  both sequences together (batch 2): 1.2525
TensorRT 11.4.0.106, builder optimization level 0: relative error against numpy
  each sequence alone (batch 1): 0.0195
  both sequences together (batch 2): 0.0195
TensorRT 11.4.0.106, builder optimization level 3: relative error against numpy
  each sequence alone (batch 1): 0.0196
  both sequences together (batch 2): 1.2525

About 0.02 is what a correct engine reaches against this numpy reference (level 0, and each sequence alone at level 3); at level 3 the batched result is off by 125%.

trt_fp4_batch_repro.py
"""Standalone TensorRT repro: an NVFP4 attention block whose result for one sequence changes with the rest of the batch.

The graph is one self-attention block as ModelOpt exports it: the input goes through TRT_FP4DynamicQuantize and two
DequantizeLinear nodes (FP8 block scales over 16 elements, then FP4 data) into the Q, K and V MatMuls, whose NVFP4
weights are initializers dequantized the same way; then the usual reshape and transposes, softmax(Q K^T / 8) V, and
the heads merged back. Random weights, no model download.

Each sequence of a batch is independent, so running a batch of 2 must give each sequence the result it gets alone.
On TensorRT 10.16 to 11.4 at builder optimization level >= 1 it does not: the engine feeds the value GEMM the FP4 input
with its rows reordered from (batch, seq) to (seq, batch) but keeps the block scales in (batch, seq) order.
usage: python trt_fp4_batch_repro.py [--level N] [--save model.onnx]
needs: tensorrt, onnx, numpy, polygraphy, an SM100/SM120 GPU
"""
from __future__ import annotations

import argparse

import numpy as np
import onnx
import tensorrt as trt
from onnx import TensorProto, helper, numpy_helper
from polygraphy.backend.trt import CreateConfig, EngineFromNetwork, NetworkFromOnnxBytes, Profile, TrtRunner

HIDDEN, HEADS, HEAD_DIM, BLOCK = 1024, 16, 64, 16
X_AMAX = 16.0
E2M1 = np.array([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0], dtype=np.float32)


def e4m3_values() -> np.ndarray:
    """Every finite non-negative FP8 E4M3 value, indexed by its bit pattern."""
    bits = np.arange(127)
    exponent, mantissa = bits >> 3, (bits & 7).astype(np.float32)
    return np.where(exponent == 0, mantissa / 8 * 2.0 ** -6, (1 + mantissa / 8) * 2.0 ** (exponent - 7.0))


E4M3 = e4m3_values()


def nearest(values: np.ndarray, grid: np.ndarray) -> np.ndarray:
    return grid[np.abs(values[..., None] - grid).argmin(axis=-1)]


def nvfp4_weight(name: str, rng: np.random.Generator) -> tuple[list[onnx.TensorProto], list[onnx.NodeProto]]:
    """A [HIDDEN, HIDDEN] NVFP4 weight: random E2M1 codes, power-of-two FP8 block scales (exact in E4M3), and a
    per-tensor scale; dequantized by DQ(block scales, per-tensor scale) -> DQ(data, scales, axis=-1, block_size=16)."""
    codes = rng.integers(0, 16, size=(HIDDEN, HIDDEN), dtype=np.uint8)  # bit 3 is the sign
    packed = (codes[:, 0::2] | (codes[:, 1::2] << 4)).astype(np.uint8)
    exponents = rng.integers(-2, 3, size=(HIDDEN, HIDDEN // BLOCK))
    scale_bytes = ((exponents + 7) << 3).astype(np.uint8)  # E4M3 bits of 2**exponent
    bias = rng.normal(0, 0.1, HIDDEN).astype(np.float16)
    values = np.where(codes & 8, -1.0, 1.0) * E2M1[codes & 7]
    WEIGHTS[name] = (values * np.repeat(2.0 ** exponents, BLOCK, axis=1) * 0.002, bias.astype(np.float32))
    initializers = [
        helper.make_tensor(f"{name}_f4", TensorProto.FLOAT4E2M1, [HIDDEN, HIDDEN], packed.tobytes(), raw=True),
        helper.make_tensor(f"{name}_sf", TensorProto.FLOAT8E4M3FN, [HIDDEN, HIDDEN // BLOCK], scale_bytes.tobytes(),
                           raw=True),
        numpy_helper.from_array(np.array(0.002, dtype=np.float16), f"{name}_gs"),
        numpy_helper.from_array(bias, f"{name}_bias"),
    ]
    nodes = [
        helper.make_node("DequantizeLinear", [f"{name}_sf", f"{name}_gs"], [f"{name}_scales"]),
        helper.make_node("DequantizeLinear", [f"{name}_f4", f"{name}_scales"], [f"{name}_w"], axis=-1,
                         block_size=BLOCK),
        helper.make_node("Transpose", [f"{name}_w"], [f"{name}_wt"], perm=[1, 0]),
    ]
    return initializers, nodes


WEIGHTS: dict[str, tuple[np.ndarray, np.ndarray]] = {}  # name -> (dequantized weight [out, in], bias)


def build_graph(rng: np.random.Generator) -> onnx.ModelProto:
    initializers = [numpy_helper.from_array(np.array(X_AMAX / 2688.0, dtype=np.float16), "x_gs"),
                    numpy_helper.from_array(np.array([0, 0, HEADS, HEAD_DIM], dtype=np.int64), "split_heads"),
                    numpy_helper.from_array(np.array([0, 0, HIDDEN], dtype=np.int64), "merge_heads"),
                    numpy_helper.from_array(np.array(HEAD_DIM ** -0.5, dtype=np.float16), "softmax_scale")]
    nodes = []
    for name in ("q", "k", "v"):
        weight_initializers, weight_nodes = nvfp4_weight(name, rng)
        initializers += weight_initializers
        nodes += weight_nodes
        nodes += [
            helper.make_node("TRT_FP4DynamicQuantize", ["x", "x_gs"], [f"{name}_x_f4", f"{name}_x_sf"], domain="trt",
                             axis=-1, block_size=BLOCK, scale_type=TensorProto.FLOAT8E4M3FN),
            helper.make_node("DequantizeLinear", [f"{name}_x_sf", "x_gs"], [f"{name}_x_scales"]),
            helper.make_node("DequantizeLinear", [f"{name}_x_f4", f"{name}_x_scales"], [f"{name}_xq"], axis=-1,
                             block_size=BLOCK),
            helper.make_node("MatMul", [f"{name}_xq", f"{name}_wt"], [f"{name}_mm"]),
            helper.make_node("Add", [f"{name}_mm", f"{name}_bias"], [f"{name}_out"]),
            helper.make_node("Reshape", [f"{name}_out", "split_heads"], [f"{name}_heads"]),
        ]
    nodes += [
        helper.make_node("Transpose", ["q_heads"], ["q_t"], perm=[0, 2, 1, 3]),
        helper.make_node("Transpose", ["k_heads"], ["k_t"], perm=[0, 2, 3, 1]),
        helper.make_node("Transpose", ["v_heads"], ["v_t"], perm=[0, 2, 1, 3]),
        helper.make_node("Mul", ["q_t", "softmax_scale"], ["q_scaled"]),
        helper.make_node("MatMul", ["q_scaled", "k_t"], ["scores"]),
        helper.make_node("Softmax", ["scores"], ["probs"], axis=-1),
        helper.make_node("MatMul", ["probs", "v_t"], ["context"]),
        helper.make_node("Transpose", ["context"], ["context_t"], perm=[0, 2, 1, 3]),
        helper.make_node("Reshape", ["context_t", "merge_heads"], ["y"]),
    ]
    graph = helper.make_graph(
        nodes, "nvfp4_attention",
        [helper.make_tensor_value_info("x", TensorProto.FLOAT16, ["batch", "seq", HIDDEN])],
        [helper.make_tensor_value_info("y", TensorProto.FLOAT16, ["batch", "seq", HIDDEN])],
        initializers)
    return helper.make_model(graph, opset_imports=[helper.make_opsetid("", 23), helper.make_opsetid("trt", 1)])


def reference(x: np.ndarray) -> np.ndarray:
    """The graph in numpy: dynamic NVFP4 of x (E4M3 block scales of amax / 6 over the per-tensor scale, E2M1 values),
    the three projections, and attention per sequence."""
    global_scale = np.float32(np.float16(X_AMAX / 2688.0))
    blocks = x.astype(np.float32).reshape(*x.shape[:-1], -1, BLOCK)
    scales = nearest(np.abs(blocks).max(axis=-1, keepdims=True) / 6 / global_scale, E4M3) * global_scale
    scaled = blocks / np.where(scales == 0, 1, scales)
    xq = (np.sign(scaled) * nearest(np.minimum(np.abs(scaled), 6), E2M1) * scales).reshape(x.shape)
    heads = {}
    for name, (weight, bias) in WEIGHTS.items():
        heads[name] = (xq @ weight.T + bias).reshape(*x.shape[:-1], HEADS, HEAD_DIM).transpose(0, 2, 1, 3)
    scores = heads["q"] @ heads["k"].transpose(0, 1, 3, 2) * HEAD_DIM ** -0.5
    probs = np.exp(scores - scores.max(axis=-1, keepdims=True))
    probs /= probs.sum(axis=-1, keepdims=True)
    return (probs @ heads["v"]).transpose(0, 2, 1, 3).reshape(x.shape)


def relative(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.linalg.norm(a - b) / np.linalg.norm(b))


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--level", type=int, default=3, help="builder optimization level")
    parser.add_argument("--save", help="also write the ONNX graph here")
    args = parser.parse_args()

    rng = np.random.default_rng(0)
    model = build_graph(rng)
    if args.save:
        onnx.save(model, args.save)
    profile = Profile().add("x", min=(1, 16, HIDDEN), opt=(4, 128, HIDDEN), max=(8, 256, HIDDEN))
    network = NetworkFromOnnxBytes(model.SerializeToString(), strongly_typed=True)
    engine = EngineFromNetwork(network, config=CreateConfig(profiles=[profile],
                                                            builder_optimization_level=args.level))
    # Tokens of different sizes, as in real activations, so that one token's block scales misfit another's values.
    token_size = rng.lognormal(0, 0.7, size=(2, 128, 1))
    x = np.clip(rng.normal(0, 1, size=(2, 128, HIDDEN)) * token_size, -X_AMAX, X_AMAX).astype(np.float16)
    with TrtRunner(engine) as runner:
        together = runner.infer({"x": x})["y"].astype(np.float32)
        alone = np.stack([runner.infer({"x": x[row:row + 1]})["y"].astype(np.float32)[0] for row in range(2)])
    expected = reference(x)
    print(f"TensorRT {trt.__version__}, builder optimization level {args.level}: relative error against numpy")
    print(f"  each sequence alone (batch 1): {relative(alone, expected):.4f}")
    print(f"  both sequences together (batch 2): {relative(together, expected):.4f}")


if __name__ == "__main__":
    main()

Have you tried the latest release?: Yes, 11.4.0.106 is affected.

Can this model run on other frameworks? ONNX Runtime does not implement TRT_FP4DynamicQuantize. The script's numpy reference implements the same graph: dynamic NVFP4 of the input, the three projections, and attention per sequence.

Langage dominant
C++
Étoiles
13.5k
Forks
2.4k
Merge moyen
6 j 21 h
PR mergées (30 j)
3

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de NVIDIA/TensorRT

Toutes les issues de NVIDIA/TensorRT

Issues similaires

Plus d'issues C++

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.