buildWtsEngine assertion when an MXFP8 block scale (E8M0) is refittable under REFIT_INDIVIDUAL

Aperta
#4,883 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 6 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
10/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp, python

Direzione di ricerca

The assertion fires in wtsEngineUtils.cpp line 123 inside the builder's weights-engine step, which the open-source tree may not contain, so first confirm where the engine code lives. Run trt_e8m0_refit_repro.py --verbose and check that only the scale_refittable case fails, with no log before the assertion. Done would mean that case builds under REFIT_INDIVIDUAL and refits correctly for a new checkpoint's block scales.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Module:Engine Build

Description

Building a REFIT_INDIVIDUAL plan (with or without STRIP_PLAN) fails when a weight of type E8M0 is marked refittable:

[TRT] [E] [wtsEngineUtils.cpp::buildWtsEngine::123] Error Code 2: Internal Error (Assertion engine failed.  In buildWtsEngine at /_src/optimizer/builder/wtsEngineUtils.cpp:123)

The main engine build succeeds ("Detected N inputs and 1 output network tensors"); the assertion is in the weights-engine step, and nothing related is logged before it, even at VERBOSE.

What narrows it down to "refittable weight of type E8M0":

  • the same E8M0 scale constant, not marked refittable, builds (and the FP8 weight it scales refits fine);
  • a float32 constant feeding Cast(E8M0) -> DQ fails the same way when marked refittable (constant folding turns it into an E8M0 weight before buildWtsEngine);
  • a float32 refittable constant behind a runtime-unfoldable op (* float32 [1,1] input == 1.0) followed by Cast(E8M0) builds, refits and runs correctly. That is the scale_workaround case in the script below; it costs a scale re-layout per GEMM at run time (~6% on a pi0.5-sized model, since the DQ is cloned per consumer).

Use case: one weight-stripped plan, refitted per fine-tuned checkpoint with that checkpoint's own MXFP8 block scales. The network is the ModelOpt MXFP8 export form (TRT_MXFP8DynamicQuantize on activations, TRT_MXFP8DequantizeLinear on an FP8 weight initializer with a UINT8 initializer holding the E8M0 exponents).

Questions:

  1. Is refitting E8M0 block scales intended to be supported under REFIT_INDIVIDUAL / STRIP_PLAN?
  2. If yes, is a fix planned for a coming release?
  3. If no, what is the recommended way to refit MXFP8 weights with per-checkpoint scales without the runtime re-layout cost of the workaround?

Environment

TensorRT Version: 10.16.1.11

NVIDIA GPU: Jetson AGX Thor (Blackwell, sm110)

NVIDIA Driver Version: 580.00 (open kernel module, aarch64; nvidia-smi reports CUDA 13.0 driver API, toolkit is 13.2)

CUDA Version: 13.2 (nvcc V13.2.78)

CUDNN Version: not used (n/a to the repro)

Operating System: JetPack 7 — L4T R38.4.0 (GCID 43443517, aarch64); Ubuntu 24.04.4 LTS inside the container

Python Version (if applicable): 3.12.3

Tensorflow Version (if applicable): n/a

PyTorch Version (if applicable): n/a (repro uses numpy + onnx + tensorrt only)

Baremetal or Container (if so, version): container, nvcr.io/nvidia/pytorch:26.05-py3 (TensorRT as shipped in that image)

Relevant Files

Model link: no model file; trt_e8m0_refit_repro.py (at the end of this report) builds the ONNX graph in memory (one MXFP8 Linear, random FP8 weight bytes and E8M0 exponents, ~100 lines).

Steps To Reproduce

Commands or scripts:

python trt_e8m0_refit_repro.py            # add --verbose for the full builder log

The script builds the same graph three times with different mark_weights_refittable sets:

case refittable weights result
weight_refittable FP8 weight OK
scale_refittable FP8 weight + UINT8/E8M0 block scale BUILD FAILED (buildWtsEngine::123)
scale_workaround FP8 weight + float32 scale behind * input(1.0) -> Cast(E8M0) OK

Output:

$ python trt_e8m0_refit_repro.py
TensorRT 10.16.1.11
=== weight_refittable: refittable = ['w_fp8']
=== weight_refittable: OK
=== scale_refittable: refittable = ['w_fp8', 'w_scale']
[10/15/2026-12:36:14] [TRT] [E] [wtsEngineUtils.cpp::buildWtsEngine::123] Error Code 2: Internal Error (Assertion engine failed.  In buildWtsEngine at /_src/optimizer/builder/wtsEngineUtils.cpp:123)
=== scale_refittable: BUILD FAILED
=== scale_workaround: refittable = ['w_fp8', 'w_scale_f32']
=== scale_workaround: OK

summary: {'weight_refittable': 'OK', 'scale_refittable': 'BUILD FAILED', 'scale_workaround': 'OK'}

Have you tried the latest release?: 10.16.1 is the latest available for Jetson Thor (JetPack 7). Not tried on x86 / newer releases.

Can this model run on other frameworks? The graph uses the trt domain ops TRT_MXFP8DynamicQuantize / TRT_MXFP8DequantizeLinear (ModelOpt MXFP8 export), so ONNX Runtime cannot run it. Not relevant to the bug: the same graph builds and runs correctly in TensorRT when the scale is not marked refittable (weight_refittable case), and the workaround case is numerically identical to it.

Repro script

trt_e8m0_refit_repro.py (click to expand)
"""Repro: TensorRT cannot build a REFIT_INDIVIDUAL plan in which an MXFP8 block scale (E8M0) is refittable.

One MXFP8 Linear in the exact form ModelOpt exports (TRT_MXFP8DynamicQuantize on the activation,
TRT_MXFP8DequantizeLinear on an FP8 weight initializer with a UINT8 block-scale initializer holding the E8M0
exponents), built with REFIT_INDIVIDUAL + STRIP_PLAN and different `mark_weights_refittable` sets:

    weight_refittable   FP8 weight marked refittable, scale baked                 -> builds
    scale_refittable    FP8 weight + scale marked refittable                      -> fails:
        [TRT] [E] [wtsEngineUtils.cpp::buildWtsEngine::123] Error Code 2: Internal Error
              (Assertion engine failed. In buildWtsEngine at /_src/optimizer/builder/wtsEngineUtils.cpp:123)
    scale_workaround    scale replaced through INetworkDefinition by
                        float32 constant (refittable) * float32 [1,1] input (=1.0) -> Cast(E8M0) -> DQ -> builds

Also fails without STRIP_PLAN, and when the scale only becomes E8M0 through constant folding (float32 constant ->
Cast(E8M0) -> DQ): any refittable weight whose type is E8M0 when `buildWtsEngine` runs asserts. The main engine build
succeeds; nothing is logged before the assertion, even at VERBOSE. The workaround costs a scale re-layout per GEMM at
run time (~6% on pi0.5). Use case: one stripped plan, refit per fine-tuned checkpoint with that checkpoint's own scales.

Seen on: TensorRT 10.16.1.11, Jetson Thor (sm110), Python API.    python trt_e8m0_refit_repro.py [--verbose]
"""

import argparse

import numpy as np
import tensorrt as trt
from onnx import TensorProto, helper

TOKENS, IN_FEATURES, OUT_FEATURES, BLOCK_SIZE = 64, 256, 128, 32
WEIGHT_NAME, SCALE_NAME, SCALE_FLOAT_NAME, ONE_INPUT_NAME = "w_fp8", "w_scale", "w_scale_f32", "one"
WEIGHT_DQ_LAYER_NAME = "node_of_w_dq"  # the parser names layers node_of_<first output>
DQ_SCALE_INPUT_INDEX = 1
E8M0_BIAS = 127
OPSET = 20
CASES = {"weight_refittable": False, "scale_refittable": True, "scale_workaround": None}


def make_onnx() -> bytes:
    rng = np.random.default_rng(0)
    weight = rng.integers(0, 0x7E, (OUT_FEATURES, IN_FEATURES), dtype=np.uint8)  # any E4M3 byte but NaN
    scale = rng.integers(E8M0_BIAS - 4, E8M0_BIAS + 4, (OUT_FEATURES, IN_FEATURES // BLOCK_SIZE), dtype=np.uint8)
    initializers = [
        helper.make_tensor(WEIGHT_NAME, TensorProto.FLOAT8E4M3FN, weight.shape, weight.tobytes(), raw=True),
        helper.make_tensor(SCALE_NAME, TensorProto.UINT8, scale.shape, scale.tobytes(), raw=True),
    ]
    mxfp8 = dict(domain="trt", axis=-1, block_size=BLOCK_SIZE)
    nodes = [
        helper.make_node("TRT_MXFP8DynamicQuantize", ["x"], ["x_fp8", "x_scale"], output_dtype=TensorProto.FLOAT8E4M3FN, **mxfp8),
        helper.make_node("TRT_MXFP8DequantizeLinear", ["x_fp8", "x_scale"], ["x_dq"], output_dtype=TensorProto.BFLOAT16, **mxfp8),
        helper.make_node("TRT_MXFP8DequantizeLinear", [WEIGHT_NAME, SCALE_NAME], ["w_dq"], output_dtype=TensorProto.BFLOAT16, **mxfp8),
        helper.make_node("Transpose", ["w_dq"], ["w_t"], perm=[1, 0]),
        helper.make_node("MatMul", ["x_dq", "w_t"], ["y"]),
    ]
    graph = helper.make_graph(
        nodes, "mxfp8_linear",
        [helper.make_tensor_value_info("x", TensorProto.BFLOAT16, [TOKENS, IN_FEATURES])],
        [helper.make_tensor_value_info("y", TensorProto.BFLOAT16, [TOKENS, OUT_FEATURES])],
        initializer=initializers,
    )
    model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", OPSET), helper.make_opsetid("trt", 1)])
    return model.SerializeToString()


def apply_workaround(network: trt.INetworkDefinition) -> None:
    """float32 constant (refittable under SCALE_FLOAT_NAME) * runtime 1.0 -> Cast(E8M0), fed to the weight DQ."""
    dq_layer = next(network.get_layer(i) for i in range(network.num_layers) if network.get_layer(i).name == WEIGHT_DQ_LAYER_NAME)
    shape = tuple(dq_layer.get_input(DQ_SCALE_INPUT_INDEX).shape)
    constant = network.add_constant(shape, trt.Weights(np.ones(shape, dtype=np.float32)))
    network.set_weights_name(constant.weights, SCALE_FLOAT_NAME)
    one = network.add_input(ONE_INPUT_NAME, trt.DataType.FLOAT, (1,) * len(shape))
    product = network.add_elementwise(constant.get_output(0), one, trt.ElementWiseOperation.PROD)
    cast = network.add_cast(product.get_output(0), trt.DataType.E8M0)
    dq_layer.set_input(DQ_SCALE_INPUT_INDEX, cast.get_output(0))


def build(case: str, onnx_bytes: bytes, trt_logger: trt.Logger) -> str:
    builder = trt.Builder(trt_logger)
    network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
    if not trt.OnnxParser(network, trt_logger).parse(onnx_bytes):
        return "PARSE FAILED"
    scale_refittable = CASES[case]
    refittable = [WEIGHT_NAME]
    if scale_refittable is None:
        apply_workaround(network)
        refittable.append(SCALE_FLOAT_NAME)
    elif scale_refittable:
        refittable.append(SCALE_NAME)
    for name in refittable:
        if not network.mark_weights_refittable(name):
            return f"mark_weights_refittable({name!r}) FAILED"

    config = builder.create_builder_config()
    config.set_flag(trt.BuilderFlag.REFIT_INDIVIDUAL)
    config.set_flag(trt.BuilderFlag.STRIP_PLAN)
    profile = builder.create_optimization_profile()
    for i in range(network.num_inputs):
        shape = list(network.get_input(i).shape)
        profile.set_shape(network.get_input(i).name, min=shape, opt=shape, max=shape)
    config.add_optimization_profile(profile)
    return "OK" if builder.build_serialized_network(network, config) is not None else "BUILD FAILED"


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--verbose", action="store_true", help="full TensorRT builder log (trt.Logger.VERBOSE)")
    args = parser.parse_args()
    trt_logger = trt.Logger(trt.Logger.VERBOSE if args.verbose else trt.Logger.WARNING)
    print(f"TensorRT {trt.__version__}")
    onnx_bytes = make_onnx()
    results = {}
    for case in CASES:
        print(f"=== {case}: refittable = {[WEIGHT_NAME] + {False: [], True: [SCALE_NAME], None: [SCALE_FLOAT_NAME]}[CASES[case]]}")
        results[case] = build(case, onnx_bytes, trt_logger)
        print(f"=== {case}: {results[case]}")
    print("\nsummary:", results)
Lingua principale
C++
Stelle
13.5k
Fork
2.4k
Merge medio
6g 21h
PR unite (30g)
3

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/TensorRT

Tutte le issue di NVIDIA/TensorRT

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.