TensorRT 10.14+ multi-GPU build regression with Cask-backed ops

オープン
#4,845 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 6 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
バグ
明瞭さ
説明が足りない
活発さ
活発
技術スタック
cpp, python

調査の方向性

まず、issue にある Python の最小再現コードを 2 つの CUDA デバイスで実行し、同一デバイスの場合とデバイスを切り替える場合の build シーケンスを比較します。2 回目の build を TensorRT の Cask ベースの MaxPool パスまで追跡します; デバイス切り替え後に 2 つ目のエンジンが正常にビルドされ、影響を受ける TensorRT バージョンと正常動作が確認されている TensorRT バージョンの両方をカバーできれば完了です。

索引モデルが issue の本文から書いたものです。

説明

Module:Engine Build

Summary

Starting with TensorRT 10.14.1, IBuilder::buildSerializedNetwork() can fail when building a second engine after switching to a different CUDA device in the same process.

The failure is specific to Cask-backed operators; the minimal repro uses MaxPool:

Error Code: 9: Skipping tactic ... Cask Pooling Runner Execute Failure
Error Code: 10: Internal Error (Could not find any implementation for node MaxPool_0.
In computeCosts at .../optimizer.cpp:4265)

The same-device sequence succeeds. Building either device alone succeeds. Building each engine in a separate process succeeds.

The regression is present in 10.14.1.48.post1 through 10.16.1.11 and is not present in 10.13.3.9 or 11.2.1.2.

Environment

  • TensorRT: tensorrt-cu13

    • 10.14.1.48.post1, 10.15.1.29, 10.16.0.72, 10.16.1.11: broken
    • 10.13.2.6, 10.13.3.9: OK
    • 11.2.1.2: OK
  • GPUs:

    • Quadro RTX 4000, SM 7.5
    • NVIDIA RTX 4000 SFF Ada Generation, SM 8.9
  • Driver: 580.142

  • OS: AlmaLinux 9.8, kernel 6.12.103-1.el9.x86_64

  • Python: 3.11

Minimal reproducer

Requires two CUDA devices.

#!/usr/bin/env python3
import ctypes
import sys

import onnx
import tensorrt as trt
from onnx import TensorProto, helper


def make_model() -> bytes:
    x = helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 1, 8, 8])
    node = helper.make_node(
        "MaxPool", ["x"], ["y"], name="MaxPool_0",
        kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[1, 1],
    )
    y = helper.make_tensor_value_info("y", TensorProto.FLOAT, [1, 1, 8, 8])
    graph = helper.make_graph([node], "tiny_maxpool", [x], [y])
    model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 20)])
    model.ir_version = 9
    onnx.checker.check_model(model)
    return model.SerializeToString()


def build(model: bytes, cudart: ctypes.CDLL, device: int, label: str) -> bool:
    assert cudart.cudaSetDevice(device) == 0
    print(f"======={label} on {device=}", file=sys.stderr)
    logger = trt.Logger(trt.Logger.INFO)
    builder = trt.Builder(logger)
    config = builder.create_builder_config()
    config.set_memory_pool_limit(
        trt.MemoryPoolType.WORKSPACE, 1 << 30
    )

    network = builder.create_network(
        1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)
    )
    parser = trt.OnnxParser(network, logger)
    assert parser.parse(model, path="/tmp")
    engine = builder.build_serialized_network(network, config)
    print("=======Failed" if engine is None else "Succeed", file=sys.stderr)


def main():
    print(f"======={trt.__version__=}", file=sys.stderr)
    cudart = ctypes.CDLL("libcudart.so")
    count = ctypes.c_int()
    cudart.cudaGetDeviceCount(ctypes.byref(count))
    print(f"======={count.value=}", file=sys.stderr)
    if count.value < 2:
        return 1

    model = make_model()

    build(model, cudart, 0, "same device 1/2")
    build(model, cudart, 0, "same device 2/2") # succeeds

    build(model, cudart, 0, "different device 1/2")
    build(model, cudart, 1, "different device 2/2") # fails after device switch

    return 0

if __name__ == "__main__":
    main()

Full traceback / output (TensorRT 10.16.1.11):

=======trt.__version__='10.16.1.11'
=======count.value=2
=======same device 1/2 on device=0
[09/05/2026-19:32:08] [TRT] [I] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 47, GPU 99 (MiB)
[09/05/2026-19:32:08] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:08] [TRT] [I] Input filename:   /tmp
[09/05/2026-19:32:08] [TRT] [I] ONNX IR version:  0.0.9
[09/05/2026-19:32:08] [TRT] [I] Opset version:    20
[09/05/2026-19:32:08] [TRT] [I] Producer name:    
[09/05/2026-19:32:08] [TRT] [I] Producer version: 
[09/05/2026-19:32:08] [TRT] [I] Domain:           
[09/05/2026-19:32:08] [TRT] [I] Model version:    0
[09/05/2026-19:32:08] [TRT] [I] Doc string:       
[09/05/2026-19:32:08] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:08] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:08] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +91, GPU +2, now: CPU 335, GPU 101 (MiB)
[09/05/2026-19:32:08] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:08] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[09/05/2026-19:32:09] [TRT] [I] Detected 1 inputs and 1 output network tensors.
[09/05/2026-19:32:09] [TRT] [I] Total Host Persistent Memory: 4160 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Device Persistent Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Max Scratch Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Activation Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Weights Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Engine generation completed in 0.412562 seconds.
[09/05/2026-19:32:09] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 1 MiB
Succeed
=======same device 2/2 on device=0
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] Input filename:   /tmp
[09/05/2026-19:32:09] [TRT] [I] ONNX IR version:  0.0.9
[09/05/2026-19:32:09] [TRT] [I] Opset version:    20
[09/05/2026-19:32:09] [TRT] [I] Producer name:    
[09/05/2026-19:32:09] [TRT] [I] Producer version: 
[09/05/2026-19:32:09] [TRT] [I] Domain:           
[09/05/2026-19:32:09] [TRT] [I] Model version:    0
[09/05/2026-19:32:09] [TRT] [I] Doc string:       
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:09] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU -89, GPU +2, now: CPU 383, GPU 103 (MiB)
[09/05/2026-19:32:09] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:09] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[09/05/2026-19:32:09] [TRT] [I] Detected 1 inputs and 1 output network tensors.
[09/05/2026-19:32:09] [TRT] [I] Total Host Persistent Memory: 4160 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Device Persistent Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Max Scratch Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Activation Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Weights Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Engine generation completed in 0.171036 seconds.
[09/05/2026-19:32:09] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 1 MiB
Succeed
=======different device 1/2 on device=0
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] Input filename:   /tmp
[09/05/2026-19:32:09] [TRT] [I] ONNX IR version:  0.0.9
[09/05/2026-19:32:09] [TRT] [I] Opset version:    20
[09/05/2026-19:32:09] [TRT] [I] Producer name:    
[09/05/2026-19:32:09] [TRT] [I] Producer version: 
[09/05/2026-19:32:09] [TRT] [I] Domain:           
[09/05/2026-19:32:09] [TRT] [I] Model version:    0
[09/05/2026-19:32:09] [TRT] [I] Doc string:       
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:09] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU -89, GPU +2, now: CPU 383, GPU 103 (MiB)
[09/05/2026-19:32:09] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:09] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[09/05/2026-19:32:09] [TRT] [I] Detected 1 inputs and 1 output network tensors.
[09/05/2026-19:32:09] [TRT] [I] Total Host Persistent Memory: 4160 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Device Persistent Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Max Scratch Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Activation Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Weights Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Engine generation completed in 0.174508 seconds.
[09/05/2026-19:32:09] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 1 MiB
Succeed
=======different device 2/2 on device=1
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] Input filename:   /tmp
[09/05/2026-19:32:09] [TRT] [I] ONNX IR version:  0.0.9
[09/05/2026-19:32:09] [TRT] [I] Opset version:    20
[09/05/2026-19:32:09] [TRT] [I] Producer name:    
[09/05/2026-19:32:09] [TRT] [I] Producer version: 
[09/05/2026-19:32:09] [TRT] [I] Domain:           
[09/05/2026-19:32:09] [TRT] [I] Model version:    0
[09/05/2026-19:32:09] [TRT] [I] Doc string:       
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU -58, GPU +0, now: CPU 413, GPU 169 (MiB)
[09/05/2026-19:32:09] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xca723f3a6d5c572a due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x1b13dea2697d92fc due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x5f085e81bccfdfd4 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xadf1751d9d52bc14 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x4805490eb56c6142 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xd4b9db5d6f22d997 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x4fc018d7ee59fb45 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x39a96deeea00c898 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x5154ad0527677a6f due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x80cf0fe785959355 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xe8f436d3bcc38289 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x31efd02f5d31b9ba due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x6abffa4236113b26 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x5faf4a0a8a5670ed due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x62f9478381204a04 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xbc27f0a63be7eb30 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xb59f9cfb90407c92 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x96dbfe2b0fabc076 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xc5cd6987d3ba33c8 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x9e9d43eab89ab154 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x9e04eb8087eb1de8 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x6cfdc01ca6767e28 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x1022263884083147 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x7213e6ffae85b15b due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x388b14a64cc134ea due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xbbe2a17f60d27137 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x6193a97a80c05353 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xf987a1a777e5f6d6 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xc17f7d05d2cb7b56 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x8cfd5f7bc7e409c2 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xbf50b5fd61a339a9 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xcd8bd446648b42ea due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x18649bf933394065 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x664b53018051029a due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xef31670ae7f6188e due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xf660830175fd03a3 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xc6ba2cdc89fee151 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x1cd68f7b324808fb due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x0b7e8a3b56789516 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x2bc9717f957e700e due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x618e02d8db64989d due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xb33a917a9f2c32a9 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x78dfe6d3496f83b0 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xb716083cd692488b due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xdfebc8d71bf5fa7c due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xe74c45c0759b786b due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x8382d5c464539e87 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xaec8628e8180bced due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xd76bac5638836f8a due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xfa211b1cdd504de0 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xe9d01a2a900075cb due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x789b2859f2e03e79 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xbd3963b8ccd084c6 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x2c7251cbae30cf74 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x22fb1bb4a70e340d due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] IBuilder::buildSerializedNetwork: Error Code 10: Internal Error (Could not find any implementation for node MaxPool_0. In computeCosts at /_src/optimizer/common/tactic/optimizer.cpp:4265)
=======Failed

Reproduction matrix

TensorRT Result
10.12.0.36 (cu12) OK
10.13.2.6 OK
10.13.3.9 OK
10.14.1.48.post1 FAIL
10.15.1.29 FAIL
10.16.0.72 FAIL
10.16.1.11 FAIL
11.2.1.2 OK

Isolation

  • Reproduces with FP32; not FP16-specific.
  • Reproduces with both 3×3 and 5×5 MaxPool.
  • Relu with the same tensor shape does not reproduce.
  • Failure follows the device switch, independent of device order.
  • cudaMemGetInfo() shows ~20 GB free on the second GPU before the build.
  • Workspace limits of 1 GB and 4 GB produce the same result.
  • cudaDeviceReset() does not resolve it.
  • IBuilderConfig.reset() does not resolve it.
  • Custom profile_stream does not resolve it.
  • Restricting/disabling tactic sources does not resolve it.
  • Dedicated OS threads do not resolve it.
  • Explicit CUDA driver contexts (cuCtxCreate/cuCtxSetCurrent) do not resolve it.
  • Building each engine in a separate OS process succeeds.

A real production model also shows the same pattern with Cask convolution execution tactic failures, suggesting the issue may affect multiple Cask-backed operators.

Expected behavior

buildSerializedNetwork() should succeed when building an engine for a different CUDA device after a previous engine was built in the same process.

Suspected cause

The behavior is consistent with process-global Cask state being initialized or cached for the first device and not correctly refreshed when the active device changes, particularly across SM 7.5 → SM 8.9.

This is a hypothesis; the minimal repro should allow the issue to be investigated independently.

Workaround

Build each GPU's engine in a separate process.

Regression range

10.13.3.9 → 10.14.1.48.post1

Fixed in 11.2.1.2.

主要言語
C++
スター
13.5k
フォーク
2.4k
平均マージ
6日 21時間
マージ済み PR(30日)
3

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/TensorRT のほかの issue

NVIDIA/TensorRT の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。