buildWtsEngine assertion when an MXFP8 block scale (E8M0) is refittable under REFIT_INDIVIDUAL

Đang mở
#4,883 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 6 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
10/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
cpp, python
Lĩnh vực
machine-learning

Hướng nghiên cứu

The assertion fires in wtsEngineUtils.cpp line 123 inside the builder's weights-engine step, which the open-source tree may not contain, so first confirm where the engine code lives. Run trt_e8m0_refit_repro.py --verbose and check that only the scale_refittable case fails, with no log before the assertion. Done would mean that case builds under REFIT_INDIVIDUAL and refits correctly for a new checkpoint's block scales.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Module:Engine Build

Description

Building a REFIT_INDIVIDUAL plan (with or without STRIP_PLAN) fails when a weight of type E8M0 is marked refittable:

[TRT] [E] [wtsEngineUtils.cpp::buildWtsEngine::123] Error Code 2: Internal Error (Assertion engine failed.  In buildWtsEngine at /_src/optimizer/builder/wtsEngineUtils.cpp:123)

The main engine build succeeds ("Detected N inputs and 1 output network tensors"); the assertion is in the weights-engine step, and nothing related is logged before it, even at VERBOSE.

What narrows it down to "refittable weight of type E8M0":

  • the same E8M0 scale constant, not marked refittable, builds (and the FP8 weight it scales refits fine);
  • a float32 constant feeding Cast(E8M0) -> DQ fails the same way when marked refittable (constant folding turns it into an E8M0 weight before buildWtsEngine);
  • a float32 refittable constant behind a runtime-unfoldable op (* float32 [1,1] input == 1.0) followed by Cast(E8M0) builds, refits and runs correctly. That is the scale_workaround case in the script below; it costs a scale re-layout per GEMM at run time (~6% on a pi0.5-sized model, since the DQ is cloned per consumer).

Use case: one weight-stripped plan, refitted per fine-tuned checkpoint with that checkpoint's own MXFP8 block scales. The network is the ModelOpt MXFP8 export form (TRT_MXFP8DynamicQuantize on activations, TRT_MXFP8DequantizeLinear on an FP8 weight initializer with a UINT8 initializer holding the E8M0 exponents).

Questions:

  1. Is refitting E8M0 block scales intended to be supported under REFIT_INDIVIDUAL / STRIP_PLAN?
  2. If yes, is a fix planned for a coming release?
  3. If no, what is the recommended way to refit MXFP8 weights with per-checkpoint scales without the runtime re-layout cost of the workaround?

Environment

TensorRT Version: 10.16.1.11

NVIDIA GPU: Jetson AGX Thor (Blackwell, sm110)

NVIDIA Driver Version: 580.00 (open kernel module, aarch64; nvidia-smi reports CUDA 13.0 driver API, toolkit is 13.2)

CUDA Version: 13.2 (nvcc V13.2.78)

CUDNN Version: not used (n/a to the repro)

Operating System: JetPack 7 — L4T R38.4.0 (GCID 43443517, aarch64); Ubuntu 24.04.4 LTS inside the container

Python Version (if applicable): 3.12.3

Tensorflow Version (if applicable): n/a

PyTorch Version (if applicable): n/a (repro uses numpy + onnx + tensorrt only)

Baremetal or Container (if so, version): container, nvcr.io/nvidia/pytorch:26.05-py3 (TensorRT as shipped in that image)

Relevant Files

Model link: no model file; trt_e8m0_refit_repro.py (at the end of this report) builds the ONNX graph in memory (one MXFP8 Linear, random FP8 weight bytes and E8M0 exponents, ~100 lines).

Steps To Reproduce

Commands or scripts:

python trt_e8m0_refit_repro.py            # add --verbose for the full builder log

The script builds the same graph three times with different mark_weights_refittable sets:

case refittable weights result
weight_refittable FP8 weight OK
scale_refittable FP8 weight + UINT8/E8M0 block scale BUILD FAILED (buildWtsEngine::123)
scale_workaround FP8 weight + float32 scale behind * input(1.0) -> Cast(E8M0) OK

Output:

$ python trt_e8m0_refit_repro.py
TensorRT 10.16.1.11
=== weight_refittable: refittable = ['w_fp8']
=== weight_refittable: OK
=== scale_refittable: refittable = ['w_fp8', 'w_scale']
[10/15/2026-12:36:14] [TRT] [E] [wtsEngineUtils.cpp::buildWtsEngine::123] Error Code 2: Internal Error (Assertion engine failed.  In buildWtsEngine at /_src/optimizer/builder/wtsEngineUtils.cpp:123)
=== scale_refittable: BUILD FAILED
=== scale_workaround: refittable = ['w_fp8', 'w_scale_f32']
=== scale_workaround: OK

summary: {'weight_refittable': 'OK', 'scale_refittable': 'BUILD FAILED', 'scale_workaround': 'OK'}

Have you tried the latest release?: 10.16.1 is the latest available for Jetson Thor (JetPack 7). Not tried on x86 / newer releases.

Can this model run on other frameworks? The graph uses the trt domain ops TRT_MXFP8DynamicQuantize / TRT_MXFP8DequantizeLinear (ModelOpt MXFP8 export), so ONNX Runtime cannot run it. Not relevant to the bug: the same graph builds and runs correctly in TensorRT when the scale is not marked refittable (weight_refittable case), and the workaround case is numerically identical to it.

Repro script

trt_e8m0_refit_repro.py (click to expand)
"""Repro: TensorRT cannot build a REFIT_INDIVIDUAL plan in which an MXFP8 block scale (E8M0) is refittable.

One MXFP8 Linear in the exact form ModelOpt exports (TRT_MXFP8DynamicQuantize on the activation,
TRT_MXFP8DequantizeLinear on an FP8 weight initializer with a UINT8 block-scale initializer holding the E8M0
exponents), built with REFIT_INDIVIDUAL + STRIP_PLAN and different `mark_weights_refittable` sets:

    weight_refittable   FP8 weight marked refittable, scale baked                 -> builds
    scale_refittable    FP8 weight + scale marked refittable                      -> fails:
        [TRT] [E] [wtsEngineUtils.cpp::buildWtsEngine::123] Error Code 2: Internal Error
              (Assertion engine failed. In buildWtsEngine at /_src/optimizer/builder/wtsEngineUtils.cpp:123)
    scale_workaround    scale replaced through INetworkDefinition by
                        float32 constant (refittable) * float32 [1,1] input (=1.0) -> Cast(E8M0) -> DQ -> builds

Also fails without STRIP_PLAN, and when the scale only becomes E8M0 through constant folding (float32 constant ->
Cast(E8M0) -> DQ): any refittable weight whose type is E8M0 when `buildWtsEngine` runs asserts. The main engine build
succeeds; nothing is logged before the assertion, even at VERBOSE. The workaround costs a scale re-layout per GEMM at
run time (~6% on pi0.5). Use case: one stripped plan, refit per fine-tuned checkpoint with that checkpoint's own scales.

Seen on: TensorRT 10.16.1.11, Jetson Thor (sm110), Python API.    python trt_e8m0_refit_repro.py [--verbose]
"""

import argparse

import numpy as np
import tensorrt as trt
from onnx import TensorProto, helper

TOKENS, IN_FEATURES, OUT_FEATURES, BLOCK_SIZE = 64, 256, 128, 32
WEIGHT_NAME, SCALE_NAME, SCALE_FLOAT_NAME, ONE_INPUT_NAME = "w_fp8", "w_scale", "w_scale_f32", "one"
WEIGHT_DQ_LAYER_NAME = "node_of_w_dq"  # the parser names layers node_of_<first output>
DQ_SCALE_INPUT_INDEX = 1
E8M0_BIAS = 127
OPSET = 20
CASES = {"weight_refittable": False, "scale_refittable": True, "scale_workaround": None}


def make_onnx() -> bytes:
    rng = np.random.default_rng(0)
    weight = rng.integers(0, 0x7E, (OUT_FEATURES, IN_FEATURES), dtype=np.uint8)  # any E4M3 byte but NaN
    scale = rng.integers(E8M0_BIAS - 4, E8M0_BIAS + 4, (OUT_FEATURES, IN_FEATURES // BLOCK_SIZE), dtype=np.uint8)
    initializers = [
        helper.make_tensor(WEIGHT_NAME, TensorProto.FLOAT8E4M3FN, weight.shape, weight.tobytes(), raw=True),
        helper.make_tensor(SCALE_NAME, TensorProto.UINT8, scale.shape, scale.tobytes(), raw=True),
    ]
    mxfp8 = dict(domain="trt", axis=-1, block_size=BLOCK_SIZE)
    nodes = [
        helper.make_node("TRT_MXFP8DynamicQuantize", ["x"], ["x_fp8", "x_scale"], output_dtype=TensorProto.FLOAT8E4M3FN, **mxfp8),
        helper.make_node("TRT_MXFP8DequantizeLinear", ["x_fp8", "x_scale"], ["x_dq"], output_dtype=TensorProto.BFLOAT16, **mxfp8),
        helper.make_node("TRT_MXFP8DequantizeLinear", [WEIGHT_NAME, SCALE_NAME], ["w_dq"], output_dtype=TensorProto.BFLOAT16, **mxfp8),
        helper.make_node("Transpose", ["w_dq"], ["w_t"], perm=[1, 0]),
        helper.make_node("MatMul", ["x_dq", "w_t"], ["y"]),
    ]
    graph = helper.make_graph(
        nodes, "mxfp8_linear",
        [helper.make_tensor_value_info("x", TensorProto.BFLOAT16, [TOKENS, IN_FEATURES])],
        [helper.make_tensor_value_info("y", TensorProto.BFLOAT16, [TOKENS, OUT_FEATURES])],
        initializer=initializers,
    )
    model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", OPSET), helper.make_opsetid("trt", 1)])
    return model.SerializeToString()


def apply_workaround(network: trt.INetworkDefinition) -> None:
    """float32 constant (refittable under SCALE_FLOAT_NAME) * runtime 1.0 -> Cast(E8M0), fed to the weight DQ."""
    dq_layer = next(network.get_layer(i) for i in range(network.num_layers) if network.get_layer(i).name == WEIGHT_DQ_LAYER_NAME)
    shape = tuple(dq_layer.get_input(DQ_SCALE_INPUT_INDEX).shape)
    constant = network.add_constant(shape, trt.Weights(np.ones(shape, dtype=np.float32)))
    network.set_weights_name(constant.weights, SCALE_FLOAT_NAME)
    one = network.add_input(ONE_INPUT_NAME, trt.DataType.FLOAT, (1,) * len(shape))
    product = network.add_elementwise(constant.get_output(0), one, trt.ElementWiseOperation.PROD)
    cast = network.add_cast(product.get_output(0), trt.DataType.E8M0)
    dq_layer.set_input(DQ_SCALE_INPUT_INDEX, cast.get_output(0))


def build(case: str, onnx_bytes: bytes, trt_logger: trt.Logger) -> str:
    builder = trt.Builder(trt_logger)
    network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
    if not trt.OnnxParser(network, trt_logger).parse(onnx_bytes):
        return "PARSE FAILED"
    scale_refittable = CASES[case]
    refittable = [WEIGHT_NAME]
    if scale_refittable is None:
        apply_workaround(network)
        refittable.append(SCALE_FLOAT_NAME)
    elif scale_refittable:
        refittable.append(SCALE_NAME)
    for name in refittable:
        if not network.mark_weights_refittable(name):
            return f"mark_weights_refittable({name!r}) FAILED"

    config = builder.create_builder_config()
    config.set_flag(trt.BuilderFlag.REFIT_INDIVIDUAL)
    config.set_flag(trt.BuilderFlag.STRIP_PLAN)
    profile = builder.create_optimization_profile()
    for i in range(network.num_inputs):
        shape = list(network.get_input(i).shape)
        profile.set_shape(network.get_input(i).name, min=shape, opt=shape, max=shape)
    config.add_optimization_profile(profile)
    return "OK" if builder.build_serialized_network(network, config) is not None else "BUILD FAILED"


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--verbose", action="store_true", help="full TensorRT builder log (trt.Logger.VERBOSE)")
    args = parser.parse_args()
    trt_logger = trt.Logger(trt.Logger.VERBOSE if args.verbose else trt.Logger.WARNING)
    print(f"TensorRT {trt.__version__}")
    onnx_bytes = make_onnx()
    results = {}
    for case in CASES:
        print(f"=== {case}: refittable = {[WEIGHT_NAME] + {False: [], True: [SCALE_NAME], None: [SCALE_FLOAT_NAME]}[CASES[case]]}")
        results[case] = build(case, onnx_bytes, trt_logger)
        print(f"=== {case}: {results[case]}")
    print("\nsummary:", results)
Ngôn ngữ chính
C++
Star
13.5k
Fork
2.4k
Merge trung bình
6 ngày 21 giờ
Pull request đã merge (30 ngày)
3

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/TensorRT

Tất cả issue của NVIDIA/TensorRT

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.