TRT 11.3: Attention does not compile inside If, even with decomposable=True

Ouverte
#4,860 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 6 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
48/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
cpp, python

Piste de recherche

Start with repro_attention_in_if_trt.py and run the failing If cases alongside --no-if and --decomposed controls. Trace parsing through trt.OnnxParser, ATTENTION_INPUT handling, and builder.build_serialized_network to isolate the If-specific failure. Done means the If model builds successfully with and without --decomposable while the existing controls remain successful.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Module:ONNX

Description

ONNX model in opset 23 of shape If { Attention } leads to internal Myelin error

[TRT] [E] Error Code: 9: Skipping tactic 0x0000000000000000 due to exception [ir_op_builder.cpp:249: myelinOpSetInput] Called with unknown input tensor or sequence name "(Unnamed Layer* 9) [ElementWise]_output". In createMyelinOp at /_src/optimizer/myelin/codeGenerator.h:1478
[TRT] [E] IBuilder::buildSerializedNetwork: Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[ONNXTRT_convertToScalar...node_of_Y_OutputLayer]}. In computeCosts at /_src/optimizer/common/tactic/optimizer.cpp:4211)

Plain Attention builds fine, only needing decomposable=True sometimes.

Environment

TensorRT Version: 11.3.0.99

NVIDIA GPU: RTX PRO 6000 Blackwell Server Edition

NVIDIA Driver Version: 580.173.02

CUDA Version: CUDA runtime 12.9 (nvidia-cuda-runtime-cu12==12.9.79, libcudart.so.12); driver libcuda.so.580.173.02

CUDNN Version: -

Operating System: Ubuntu 20.04.6 pod on kernel 6.18

Python Version (if applicable): 3.11.16

Tensorflow Version (if applicable): -

PyTorch Version (if applicable): -

Steps To Reproduce

import argparse
import sys

import numpy as np
import onnx
from onnx import TensorProto, helper

try:
    import tensorrt as trt
except ImportError:
    import tensorrt_bindings as trt

ap = argparse.ArgumentParser()
ap.add_argument("--decomposable", action="store_true")
ap.add_argument("--else", dest="else_op", default="identity", choices=["identity", "mul"])
ap.add_argument("--no-if", action="store_true", help="control: Attention at top level, no If")
ap.add_argument("--decomposed", action="store_true", help="control: opset-18 MatMul/Softmax form instead of Attention")
ap.add_argument("--dtype", default="fp32", choices=["fp32", "fp16"])
ap.add_argument("--shape", default="1,1,4,8")
args = ap.parse_args()

elem = TensorProto.FLOAT if args.dtype == "fp32" else TensorProto.FLOAT16
np_dtype = np.float32 if args.dtype == "fp32" else np.float16
shape = [int(s) for s in args.shape.split(",")]
opset = 18 if args.decomposed else 23


def attention_nodes(out: str) -> list:
    if not args.decomposed:
        return [helper.make_node("Attention", ["Q", "K", "V"], [out])]
    return [
        helper.make_node("Transpose", ["K"], ["KT"], perm=[0, 1, 3, 2]),
        helper.make_node("MatMul", ["Q", "KT"], ["S"]),
        helper.make_node("Mul", ["S", "scale"], ["SS"]),
        helper.make_node("Softmax", ["SS"], ["P"], axis=-1),
        helper.make_node("MatMul", ["P", "V"], [out]),
    ]


initializers = [helper.make_tensor("scale", elem, [], np.array([1.0 / np.sqrt(shape[-1])], dtype=np_dtype).tolist()),
                helper.make_tensor("two", elem, [], np.array([2.0], dtype=np_dtype).tolist())]
qkv = [helper.make_tensor_value_info(n, elem, shape) for n in ("Q", "K", "V")]
y = helper.make_tensor_value_info("Y", elem, shape)

if args.no_if:
    graph = helper.make_graph(attention_nodes("Y"), "attn", qkv, [y], initializer=initializers)
else:
    cond = helper.make_tensor_value_info("cond", TensorProto.BOOL, [1])
    then_g = helper.make_graph(attention_nodes("Yt"), "then", [], [helper.make_tensor_value_info("Yt", elem, shape)])
    else_node = (helper.make_node("Identity", ["Q"], ["Ye"]) if args.else_op == "identity"
                 else helper.make_node("Mul", ["Q", "two"], ["Ye"]))
    else_g = helper.make_graph([else_node], "else", [], [helper.make_tensor_value_info("Ye", elem, shape)])
    if_node = helper.make_node("If", ["cond"], ["Y"], then_branch=then_g, else_branch=else_g)
    graph = helper.make_graph([if_node], "attn_in_if", [cond, *qkv], [y], initializer=initializers)

model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", opset)], ir_version=10)
onnx.checker.check_model(model)

logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(0)
parser = trt.OnnxParser(network, logger)
if not parser.parse(model.SerializeToString(), "attention_in_if.onnx"):
    sys.exit("\n".join(str(parser.get_error(i)) for i in range(parser.num_errors)))

n_attn = 0
for i in range(network.num_layers):
    layer = network.get_layer(i)
    if layer.type == trt.LayerType.ATTENTION_INPUT:
        n_attn += 1
        if args.decomposable:
            layer.__class__ = trt.IAttentionInputLayer
            layer.attention.decomposable = True

config = builder.create_builder_config()
config.clear_flag(trt.BuilderFlag.TF32)
plan = builder.build_serialized_network(network, config)
print(f"tensorrt {trt.__version__} if={not args.no_if} else={args.else_op} decomposed_onnx={args.decomposed} "
      f"dtype={args.dtype} shape={shape} attention_layers={n_attn} decomposable={args.decomposable}: "
      + ("BUILD OK" if plan is not None else "BUILD FAILED"), flush=True)
sys.exit(0 if plan is not None else 1)

Commands or scripts:
Save provided script as repro_attention_in_if_trt.py
Bug is reproduced when you run it with and without --decomposable

Controls:
pre-decomposed --decomposed works fine (opset 18 does not have Attention)
without if --no-if works fine, maybe requiring --decomposable

Have you tried the latest release?: yes

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

Langage dominant
C++
Étoiles
13.5k
Forks
2.4k
Merge moyen
6 j 21 h
PR mergées (30 j)
3

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de NVIDIA/TensorRT

Toutes les issues de NVIDIA/TensorRT

Issues similaires

Plus d'issues C++

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.