[TensorRT 11.x][Blackwell SM120] Internal DDS compiler assertion when IPluginV3 scalar output controls dynamic ONNX Slice

オープン
#4,878 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 6 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
5/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
cpp, shell

調査の方向性

The failing assertion is in convertExplicitDDSPluginToImplicit.cpp at line 149, inside the TensorRT myelin optimizer, and the report reproduces it only with a proprietary IPluginV3 plugin (SparseConvolution) feeding two dynamic ONNX Slice ops. Reproducing needs a Blackwell SM120 GPU, TensorRT 11.x, and a substitute plugin that declares a scalar INT32 output without calling IExprBuilder::declareSizeTensor(). Done would mean understanding why the DDS output and size-tensor bookkeeping disagree, which is maintainer-level work on the closed-source optimizer.

索引モデルが issue の本文から書いたものです。

説明

Module:Engine Build

Description

We are experiencing what seems an architecture and version-dependent TensorRT engine compilation failure involving a custom IPluginV3 and downstream data-dependent ONNX Slice operations.

The compilation from onnx to engine is done with trtexec.

The behavior differs depending on the GPU architecture and TensorRT version:

  • TensorRT 10.14: Successfully builds the engine on Blackwell SM120.
  • TensorRT 11.1 and 11.4: Fail during engine compilation on Blackwell SM120.
  • TensorRT 11.1 and 11.4: Successfully build the same network on Ampere SM86 and Ada SM89 GPUs (only tested with 11.1).

The original model and custom plugin implementation are proprietary and cannot be shared.

Environment

GPU Compute capability TensorRT 10.14 TensorRT 11.1 TensorRT 11.4
NVIDIA RTX A3000 Laptop SM86 PASS PASS PASS
NVIDIA RTX 4090 SM89 PASS PASS NOT TESTED
NVIDIA RTX PRO 4000 Blackwell SFF SM120 PASS FAIL FAIL

Additional environment information:

  • ONNX opset: 17
  • Custom plugin interface: IPluginV3
  • Engine building: trtexec
  • Plugin loading: --staticPlugins
  • Optimization profile: dynamic input dimensions

Note that in 10.14 we were using implicit trtexec fp32 -> fp16 conversion

Exact Error

The engine build fails with the following internal assertion:

[E] Error[2]:
[convertExplicitDDSPluginToImplicit.cpp::
convertExplicitDDSPluginToImplicit::149]

Error Code 2: Internal Error

Assertion
nodeIdxToDDSOutputIndices.count(i) ==
nodeIdxToSizeTensors.contains(i)
failed.

In convertExplicitDDSPluginToImplicit at
/_src/optimizer/myelin/rewrite/
convertExplicitDDSPluginToImplicit.cpp:149

The error occurs during engine construction, not during inference.

Steps To Reproduce

The relevant graph section consists of a custom IPluginV3 (SparseConvolution) producing fixed-capacity tensors and a scalar indicating how many entries are valid.

The plugin exposes three outputs:

Output Data type Shape Description
0 FP16 [CAPACITY, CHANNELS] Feature data
1 INT32 [CAPACITY, 4] Associated indices
2 INT32 [] Runtime number of valid entries

The scalar is subsequently used to determine the output dimension of two ONNX Slice operations.

Conceptually, the operation is equivalent to:

valid_features = features[:valid_count]
valid_indices = indices[:valid_count]

The count is a runtime value produced by the custom plugin.
The plugin currently declares the count as an ordinary INT32 scalar output and does not explicitly call IExprBuilder::declareSizeTensor().

We would like to confirm whether this pattern is supported by TensorRT 11.x. or a regression on TensorRT 11. on Blackwell path

主要言語
C++
スター
13.5k
フォーク
2.4k
平均マージ
6日 21時間
マージ済み PR(30日)
3

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/TensorRT のほかの issue

NVIDIA/TensorRT の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。