Myelin NVRTC compilation failure on Jetson Thor (SM110) building Fast-FoundationStereo - reproduces on TRT 10.14.1 and 11.2.1.2

未关闭
#4,830 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 6 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
25/100
Issue 类型
缺陷
描述清晰度
需要澄清
活跃度
冷清
技术栈
cpp, python

调研方向

首先,使用 trtexec 或 Python API,通过 scripts/make_single_onnx.py 的较新 ONNX 导出复现该失败,然后将其与之前正常工作的导出进行比较。阅读相关 issue #4715,并检查报告中提到的 make_plugin_onnx.py 和 C++ plugin 路径。确认存在 fix 或 workaround,并记录预期的 fp16 和 plugin 部署行为后,即视为完成。

由索引模型根据 Issue 内容生成。

描述

Module:Embedded Module:Engine Build

Title: Myelin NVRTC compilation failure on Jetson Thor (SM110) building Fast-FoundationStereo — reproduces on TRT 10.14.1 and 11.2.1.2

Environment

  • Jetson Thor (SM110), JetPack 7 / L4T, unified memory 122 GB
  • TensorRT 10.14.1 (JetPack tarball trtexec) and 11.2.1.2 (pip tensorrt cu13 aarch64, python API)
  • Model: Fast-FoundationStereo, exported from the NGC checkpoint (nvidia/tao/fast-foundationstereo v1.2, model_best_bp2_serialize.pth) with the official scripts/make_single_onnx.py (NVlabs/Fast-FoundationStereo, opset 17), --height 288 --width 480 --valid_iters 4, exporter run in nvcr.io/nvidia/pytorch:25.06-py3-igpu

Failure (identical on both TRT versions, fp32 — no precision flags involved)

TRT 10.14.1:

Error[9]: Skipping tactic 0x0 due to exception [myelin_graph.h:1182] MyelinCheckException:
nvrtc_compile.cpp:1118: CHECK(success) failed. NVRTC Compilation failure
Error[10]: IBuilder::buildSerializedNetworkToStream: Error Code 10: Internal Error
(Could not find any implementation for node {ForeignNode[/Concat.../Unsqueeze_166]}.)

TRT 11.2.1.2:

MyelinCheckException: nvrtc_compile.cpp:1122: CHECK(success) failed. NVRTC Compilation failure
Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[/Concat.../Cast_118]}.)

Additional data points

  • An older single-ONNX export of the same model (576x960, iters_8, exported ~May 2026 with an older torch) builds and runs correctly in fp32 on the same machine/TRT 10.14.1 (1199 ms) — the failure is specific to graph patterns emitted by the newer exporter/torch.
  • fp16 builds of the older export segfault the TRT compiler on SM110 (10.13.3 and 10.14.1; July 2026 finding).
  • Full FoundationStereo-S (NGC dynamic ONNX) at 288x480 builds fp32 on 10.14.1 (1052 ms) — so small dims per se are fine.
  • Looks related to #4715 (sm_120: Myelin "no valid tactics" on Fast-FS post_runner at all precisions, fused 3D-ConvTranspose/Cast pattern).

Questions

  1. Is there a fix or workaround for the SM110 Myelin/NVRTC codegen failure (builder flags, tactic source restrictions, graph rewrites)?
  2. Is the repo's plugin path (make_plugin_onnx.py + cpp plugin) the intended deployment route on Thor today?
  3. Is fp16 expected to work for this model on SM110, given the exporter's own printed recipe suggests trtexec --fp16?

Full build logs and both ONNX files available on request.

主要语言
C++
星标
13.5k
派生
2.4k
平均合并
6 天 21 小时
30 天内合并 PR
3

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

NVIDIA/TensorRT 的其他 Issue

查看 NVIDIA/TensorRT 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。