Myelin exec_with_offset failure at run time with builder_optimization_level=4 for part of a dynamic-batch profile (TensorRT 10.16, H100)

未关闭
#4,898 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 6 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
30/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
python

调研方向

Start by reproducing the failure with the supplied ONNX model, TensorRT 10.16, and H100, probing batch sizes 146–256 in profile 1 at builder optimization level 4; compare with level 3. The report does not include a minimal plugin-free reproducer or kernel-level evidence, so first isolate whether the custom plugin is involved and inspect the failing tactic. Done means identifying a reproducible cause or confirming a TensorRT limitation and documenting the result.

由索引模型根据 Issue 内容生成。

描述

Description

With builder_optimization_level = 4, an FP16 engine with three optimisation profiles builds fine but fails at run time for part of the middle profile. enqueueV3 returns false and logs:

[E] IExecutionContext::enqueueV3: Error Code 1: Myelin ([cask.cpp:1433: exec_with_offset] Platform (Cuda) error In executeMyelinGraph at /_src/runtime/myelin/runner.cpp:878)

The same ONNX built at the default level 3 runs every batch size. At level 4 the engine is 4–10% faster where it does run.

What fails

One dynamic batch dimension, 3 optimisation profiles (min,opt,max): (1,16,32), (33,128,256), (257,512,1024). We ran the engine once at every batch size 1–1024, synchronising after each run, in two ways: through the per-profile execution contexts, and through a fresh fixed-shape context per batch (what CUDA-graph capture uses). Two variants of the network (different input sizes) behave the same:

Profile 0 (1–32) Profile 1 (33–256) Profile 2 (257–1,024)
Variant A 0 / 32 fail 110 / 224 fail, B = 147–256 0 / 768 fail
Variant B 0 / 32 fail 106 / 224 fail, B = 151–256 0 / 768 fail
  • Eager and bound-context runs fail on the same batch sizes, so CUDA-graph capture is not the cause.
  • The error is not sticky: the CUDA context survives each failure and later batch sizes still run.
  • Same ONNX at level 3: 0 failures across 1–1024.

Guess at the cause (not confirmed)

Failures start between B = 146 and B = 147 in a profile with opt shape 128. That is consistent with a launch dimension of about B × 448 crossing CUDA's 65,535 grid limit, but we have not confirmed it at the kernel level. It looks as if the level-4 tactic chosen for profile 1 is not valid for the whole profile range.

Environment

  • TensorRT 10.16 (Python wheel)
  • NVIDIA H100 80 GB (SM90)
  • Driver and CUDA runtime versions were not recorded for these runs.
  • The network also contains a custom plugin (QDP API, ahead-of-time compiled). We have not tried to reproduce the failure without it, so it may be involved. A minimal plugin-free reproducer is not available yet.

Workaround

Serving batches 1–256 from a level-3 engine and 257–1,024 from the level-4 engine avoids the failing range.

Request

Is this a known limitation of level-4 tactic selection for dynamic profiles? I can run a per-batch-size probe on a newer build if that helps.

主要语言
C++
星标
13.5k
派生
2.4k
平均合并
6 天 21 小时
30 天内合并 PR
3

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

NVIDIA/TensorRT 的其他 Issue

查看 NVIDIA/TensorRT 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。