Myelin exec_with_offset failure at run time with builder_optimization_level=4 for part of a dynamic-batch profile (TensorRT 10.16, H100)

Aperta
#4,898 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 6 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python

Direzione di ricerca

Start by reproducing the failure with the supplied ONNX model, TensorRT 10.16, and H100, probing batch sizes 146–256 in profile 1 at builder optimization level 4; compare with level 3. The report does not include a minimal plugin-free reproducer or kernel-level evidence, so first isolate whether the custom plugin is involved and inspect the failing tactic. Done means identifying a reproducible cause or confirming a TensorRT limitation and documenting the result.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Description

With builder_optimization_level = 4, an FP16 engine with three optimisation profiles builds fine but fails at run time for part of the middle profile. enqueueV3 returns false and logs:

[E] IExecutionContext::enqueueV3: Error Code 1: Myelin ([cask.cpp:1433: exec_with_offset] Platform (Cuda) error In executeMyelinGraph at /_src/runtime/myelin/runner.cpp:878)

The same ONNX built at the default level 3 runs every batch size. At level 4 the engine is 4–10% faster where it does run.

What fails

One dynamic batch dimension, 3 optimisation profiles (min,opt,max): (1,16,32), (33,128,256), (257,512,1024). We ran the engine once at every batch size 1–1024, synchronising after each run, in two ways: through the per-profile execution contexts, and through a fresh fixed-shape context per batch (what CUDA-graph capture uses). Two variants of the network (different input sizes) behave the same:

Profile 0 (1–32) Profile 1 (33–256) Profile 2 (257–1,024)
Variant A 0 / 32 fail 110 / 224 fail, B = 147–256 0 / 768 fail
Variant B 0 / 32 fail 106 / 224 fail, B = 151–256 0 / 768 fail
  • Eager and bound-context runs fail on the same batch sizes, so CUDA-graph capture is not the cause.
  • The error is not sticky: the CUDA context survives each failure and later batch sizes still run.
  • Same ONNX at level 3: 0 failures across 1–1024.

Guess at the cause (not confirmed)

Failures start between B = 146 and B = 147 in a profile with opt shape 128. That is consistent with a launch dimension of about B × 448 crossing CUDA's 65,535 grid limit, but we have not confirmed it at the kernel level. It looks as if the level-4 tactic chosen for profile 1 is not valid for the whole profile range.

Environment

  • TensorRT 10.16 (Python wheel)
  • NVIDIA H100 80 GB (SM90)
  • Driver and CUDA runtime versions were not recorded for these runs.
  • The network also contains a custom plugin (QDP API, ahead-of-time compiled). We have not tried to reproduce the failure without it, so it may be involved. A minimal plugin-free reproducer is not available yet.

Workaround

Serving batches 1–256 from a level-3 engine and 257–1,024 from the level-4 engine avoids the failing range.

Request

Is this a known limitation of level-4 tactic selection for dynamic profiles? I can run a per-batch-size probe on a newer build if that helps.

Lingua principale
C++
Stelle
13.5k
Fork
2.4k
Merge medio
6g 21h
PR unite (30g)
3

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/TensorRT

Tutte le issue di NVIDIA/TensorRT

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.