Myelin exec_with_offset failure at run time with builder_optimization_level=4 for part of a dynamic-batch profile (TensorRT 10.16, H100)

Offen
#4,898 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 6 Tagen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
30/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python

Rechercherichtung

Start by reproducing the failure with the supplied ONNX model, TensorRT 10.16, and H100, probing batch sizes 146–256 in profile 1 at builder optimization level 4; compare with level 3. The report does not include a minimal plugin-free reproducer or kernel-level evidence, so first isolate whether the custom plugin is involved and inspect the failing tactic. Done means identifying a reproducible cause or confirming a TensorRT limitation and documenting the result.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Description

With builder_optimization_level = 4, an FP16 engine with three optimisation profiles builds fine but fails at run time for part of the middle profile. enqueueV3 returns false and logs:

[E] IExecutionContext::enqueueV3: Error Code 1: Myelin ([cask.cpp:1433: exec_with_offset] Platform (Cuda) error In executeMyelinGraph at /_src/runtime/myelin/runner.cpp:878)

The same ONNX built at the default level 3 runs every batch size. At level 4 the engine is 4–10% faster where it does run.

What fails

One dynamic batch dimension, 3 optimisation profiles (min,opt,max): (1,16,32), (33,128,256), (257,512,1024). We ran the engine once at every batch size 1–1024, synchronising after each run, in two ways: through the per-profile execution contexts, and through a fresh fixed-shape context per batch (what CUDA-graph capture uses). Two variants of the network (different input sizes) behave the same:

Profile 0 (1–32) Profile 1 (33–256) Profile 2 (257–1,024)
Variant A 0 / 32 fail 110 / 224 fail, B = 147–256 0 / 768 fail
Variant B 0 / 32 fail 106 / 224 fail, B = 151–256 0 / 768 fail
  • Eager and bound-context runs fail on the same batch sizes, so CUDA-graph capture is not the cause.
  • The error is not sticky: the CUDA context survives each failure and later batch sizes still run.
  • Same ONNX at level 3: 0 failures across 1–1024.

Guess at the cause (not confirmed)

Failures start between B = 146 and B = 147 in a profile with opt shape 128. That is consistent with a launch dimension of about B × 448 crossing CUDA's 65,535 grid limit, but we have not confirmed it at the kernel level. It looks as if the level-4 tactic chosen for profile 1 is not valid for the whole profile range.

Environment

  • TensorRT 10.16 (Python wheel)
  • NVIDIA H100 80 GB (SM90)
  • Driver and CUDA runtime versions were not recorded for these runs.
  • The network also contains a custom plugin (QDP API, ahead-of-time compiled). We have not tried to reproduce the failure without it, so it may be involved. A minimal plugin-free reproducer is not available yet.

Workaround

Serving batches 1–256 from a level-3 engine and 257–1,024 from the level-4 engine avoids the failing range.

Request

Is this a known limitation of level-4 tactic selection for dynamic profiles? I can run a per-batch-size probe on a newer build if that helps.

Vorherrschende Sprache
C++
Sterne
13.5k
Forks
2.4k
Ø Merge
6 T. 21 Std.
Gemergte PRs (30 T.)
3

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus NVIDIA/TensorRT

Alle Issues in NVIDIA/TensorRT

Ähnliche Issues

Weitere Issues zu C++

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.