Myelin exec_with_offset failure at run time with builder_optimization_level=4 for part of a dynamic-batch profile (TensorRT 10.16, H100)
维护者通常 6 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 30/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- python
调研方向
Start by reproducing the failure with the supplied ONNX model, TensorRT 10.16, and H100, probing batch sizes 146–256 in profile 1 at builder optimization level 4; compare with level 3. The report does not include a minimal plugin-free reproducer or kernel-level evidence, so first isolate whether the custom plugin is involved and inspect the failing tactic. Done means identifying a reproducible cause or confirming a TensorRT limitation and documenting the result.
由索引模型根据 Issue 内容生成。
描述
Description
With builder_optimization_level = 4, an FP16 engine with three optimisation profiles builds fine but fails at run time for part of the middle profile. enqueueV3 returns false and logs:
[E] IExecutionContext::enqueueV3: Error Code 1: Myelin ([cask.cpp:1433: exec_with_offset] Platform (Cuda) error In executeMyelinGraph at /_src/runtime/myelin/runner.cpp:878)
The same ONNX built at the default level 3 runs every batch size. At level 4 the engine is 4–10% faster where it does run.
What fails
One dynamic batch dimension, 3 optimisation profiles (min,opt,max): (1,16,32), (33,128,256), (257,512,1024). We ran the engine once at every batch size 1–1024, synchronising after each run, in two ways: through the per-profile execution contexts, and through a fresh fixed-shape context per batch (what CUDA-graph capture uses). Two variants of the network (different input sizes) behave the same:
| Profile 0 (1–32) | Profile 1 (33–256) | Profile 2 (257–1,024) | |
|---|---|---|---|
| Variant A | 0 / 32 fail | 110 / 224 fail, B = 147–256 | 0 / 768 fail |
| Variant B | 0 / 32 fail | 106 / 224 fail, B = 151–256 | 0 / 768 fail |
- Eager and bound-context runs fail on the same batch sizes, so CUDA-graph capture is not the cause.
- The error is not sticky: the CUDA context survives each failure and later batch sizes still run.
- Same ONNX at level 3: 0 failures across 1–1024.
Guess at the cause (not confirmed)
Failures start between B = 146 and B = 147 in a profile with opt shape 128. That is consistent with a launch dimension of about B × 448 crossing CUDA's 65,535 grid limit, but we have not confirmed it at the kernel level. It looks as if the level-4 tactic chosen for profile 1 is not valid for the whole profile range.
Environment
- TensorRT 10.16 (Python wheel)
- NVIDIA H100 80 GB (SM90)
- Driver and CUDA runtime versions were not recorded for these runs.
- The network also contains a custom plugin (QDP API, ahead-of-time compiled). We have not tried to reproduce the failure without it, so it may be involved. A minimal plugin-free reproducer is not available yet.
Workaround
Serving batches 1–256 from a level-3 engine and 257–1,024 from the level-4 engine avoids the failing range.
Request
Is this a known limitation of level-4 tactic selection for dynamic profiles? I can run a per-batch-size probe on a newer build if that helps.
- 主要语言
- C++
- 星标
- 13.5k
- 派生
- 2.4k
- 平均合并
- 6 天 21 小时
- 30 天内合并 PR
- 3
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
NVIDIA/TensorRT 的其他 Issue
-
Module:Polygraphy
难度 2/5 1-3 小时 新手友好度 77/100
维护者通常 6 天内回复
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectly可能已有人在做 @0z5a 于 16 天前认领。 未关闭Module:Polygraphy
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 6 天内回复
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerance可能已有人在做 @0z5a 于 16 天前认领。 未关闭Module:Polygraphy
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 6 天内回复
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop node可能已有人在做 @gabortar 于 39 天前认领。 未关闭Module:GraphSurgeon
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 6 天内回复
-
Polygraphy parse_meta accepts a shape with a missing bracket可能已有人在做 @SashaMIT 于 51 天前认领。 未关闭Module:Polygraphy
难度 2/5 1-3 小时 新手友好度 86/100
维护者通常 6 天内回复
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 91/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 69/100
maplibre/maplibre-native#5072 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 79/100
tenstorrent/tt-umd#4017 ·
维护者通常 3 天内回复
-
difficulty:easy func:gui prio:low type:bug
难度 2/5 1-3 小时 新手友好度 72/100
NatronGitHub/Natron#1538 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
hyprwm/hyprsunset#103 · 1 条评论 ·