ScatterND in a dynamic-shape engine silently drops updates beyond ~N when the update count M exceeds the output size N (static engine correct)
Los mantenedores suelen responder en 6 días
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 55/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- python
- Área
- machine-learning
Línea de trabajo
Start by running the reproduction script from the issue and compare the dynamic and static engine results with the ONNX reference evaluator. Investigate the dynamic-shape ScatterND execution path, using the reported N/M variations and GPU results to narrow the cutoff. Done means all 1000 real updates are applied for dynamic shapes when M exceeds N, without silent drops or warnings.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
ScatterND in a dynamic-shape engine silently drops updates beyond ~N when the update count M exceeds the
output size N (static engine correct)
Environment: TensorRT 11.2.1.2 (tensorrt-cu13 Python wheels), CUDA runtime 13.0.96, driver 580.142; Quadro RTX
4000 (sm_75) and RTX 4000 SFF Ada Generation (sm_89); Linux (AlmaLinux 9.8), Python 3.14; ONNX opset 18, strongly
typed network, one optimization profile.
Description: A single ONNX ScatterND (data (N,), indices (M, 1), updates (M,)) built into an engine with a
dynamic-shape optimization profile applies only about the first N entries of the update list when M > N; later
entries are silently not written. The same graph built as a static engine, onnxruntime, and the ONNX reference
evaluator apply all updates. With M <= N the dynamic engine is correct, even with the same duplicate indices.
Making either N or M dynamic is enough. fp32 and fp16 behave the same; both GPUs above are affected.
The update list in the repro contains duplicate indices (all non-test updates write index N - 1), which the ONNX
spec says ScatterND indices should not have, because the iteration order is unspecified. The updates that get
dropped, however, target indices that occur only once, so any iteration order would write them; the result differs
between static and dynamic engines of the same graph, and depends on M vs N. There is no build- or run-time
warning. If duplicates make this undefined in TensorRT, a warning or an error would help; we lost masks in
production-like output without any signal.
Steps to reproduce: run the script below (needs tensorrt, onnx, numpy, and torch with CUDA for device
buffers).
"""ScatterND in a dynamic-shape engine drops
updates past about N when M > N (with M <= N, the same duplicates are fine).
One ScatterND: data (N,) zeros, indices (M, 1), updates (M,). 1000 "real" updates with unique indices 0..999
(values 1..1000) are spread evenly over the update list; every other update writes index N-1. Expected (ONNX
reference / onnxruntime / a static TensorRT engine): out[0:1000] == 1..1000.
Needs: tensorrt, onnx, numpy, torch (CUDA, only for device buffers).
"""
import numpy as np
import tensorrt as trt
import torch
from onnx import TensorProto
from onnx import helper
from onnx.reference import ReferenceEvaluator
N, M, REAL = 91585, 144000, 1000
def model(dynamic: bool) -> bytes:
n, m = ("N", "M") if dynamic else (N, M)
g = helper.make_graph(
[helper.make_node("ScatterND", ["data", "indices", "updates"], ["out"])], "scatternd",
[helper.make_tensor_value_info("data", TensorProto.FLOAT, [n]),
helper.make_tensor_value_info("indices", TensorProto.INT64, [m, 1]),
helper.make_tensor_value_info("updates", TensorProto.FLOAT, [m])],
[helper.make_tensor_value_info("out", TensorProto.FLOAT, [n])])
mdl = helper.make_model(g, opset_imports=[helper.make_opsetid("", 18)])
mdl.ir_version = 9
return mdl.SerializeToString()
def run(dynamic: bool, data, indices, updates) -> np.ndarray:
logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
parser = trt.OnnxParser(network, logger)
assert parser.parse(model(dynamic))
config = builder.create_builder_config()
if dynamic:
p = builder.create_optimization_profile()
p.set_shape("data", (N,), (2 * N,), (6 * N,))
p.set_shape("indices", (M, 1), (2 * M, 1), (15 * M, 1))
p.set_shape("updates", (M,), (2 * M,), (15 * M,))
config.add_optimization_profile(p)
engine = trt.Runtime(logger).deserialize_cuda_engine(builder.build_serialized_network(network, config))
ctx = engine.create_execution_context()
bufs = {"data": torch.from_numpy(data).cuda(), "indices": torch.from_numpy(indices).cuda(),
"updates": torch.from_numpy(updates).cuda(), "out": torch.empty(len(data), device="cuda")}
for name in ("data", "indices", "updates"):
ctx.set_input_shape(name, tuple(bufs[name].shape))
for name, t in bufs.items():
ctx.set_tensor_address(name, t.data_ptr())
stream = torch.cuda.Stream()
ctx.execute_async_v3(stream.cuda_stream)
stream.synchronize()
return bufs["out"].cpu().numpy()
def inputs(n: int) -> tuple[np.ndarray, ...]:
"""1000 real updates (unique indices 0..999) spread over the M-long update list; the rest write index n - 1."""
positions = np.linspace(0, M - 1, REAL).astype(np.int64)
indices = np.full((M, 1), n - 1, np.int64)
indices[positions, 0] = np.arange(REAL)
updates = np.full(M, -1.0, np.float32)
updates[positions] = np.arange(1, REAL + 1)
return np.zeros(n, np.float32), indices, updates, positions
expected = np.arange(1, REAL + 1, dtype=np.float32)
print(f"TensorRT {trt.__version__}, {torch.cuda.get_device_name()}; M = {M} updates")
data, indices, updates, positions = inputs(N)
(reference,) = ReferenceEvaluator(model(dynamic=False)).run(None, {"data": data, "indices": indices, "updates": updates})
print(f" N = {N} (M > N), ONNX reference: {int((reference[:REAL] == expected).sum())}/{REAL} real updates applied")
for dynamic, n in ((False, N), (True, N), (True, 2 * N)): # the last: the same duplicates, but M <= N
data, indices, updates, positions = inputs(n)
ok = run(dynamic, data, indices, updates)[:REAL] == expected
first_dropped = positions[~ok].min() if (~ok).any() else None
print(f" N = {n} ({'M > N' if n < M else 'M <= N'}), {'dynamic' if dynamic else 'static '} engine: {int(ok.sum())}/{REAL} real updates applied"
+ ("" if first_dropped is None else f"; first dropped at update-list position {first_dropped}"))
Expected: every line reports 1000 / 1000 real updates applied.
Actual (both GPUs; the cutoff differs slightly):
TensorRT 11.2.1.2, NVIDIA RTX 4000 SFF Ada Generation; M = 144000 updates
N = 91585 (M > N), ONNX reference: 1000/1000 real updates applied
N = 91585 (M > N), static engine: 1000/1000 real updates applied
N = 91585 (M > N), dynamic engine: 636/1000 real updates applied; first dropped at update-list position 91675
N = 183170 (M <= N), dynamic engine: 1000/1000 real updates applied
TensorRT 11.2.1.2, Quadro RTX 4000; M = 144000 updates
N = 91585 (M > N), ONNX reference: 1000/1000 real updates applied
N = 91585 (M > N), static engine: 1000/1000 real updates applied
N = 91585 (M > N), dynamic engine: 640/1000 real updates applied; first dropped at update-list position 92251
N = 183170 (M <= N), dynamic engine: 1000/1000 real updates applied
Additional observations: the applied updates are always a prefix of the update list, ending within 0.1-0.7 % of
N, across N = 91,585 - 547,244 and M = 144,000 - 2,112,000 in one profile; only M dynamic (N static) or only N
dynamic (M static) reproduce it too.
- Lenguaje dominante
- C++
- Estrellas
- 13.5k
- Forks
- 2.4k
- Merge medio
- 6 d 21 h
- PR fusionados (30 d)
- 3
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de NVIDIA/TensorRT
-
Module:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 77/100
Los mantenedores suelen responder en 6 días
-
Polygraphy NumPy Top-K sorts unsigned and minimum signed integers incorrectlyPosiblemente ocupada @0z5a la tomó hace 16 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 6 días
-
Polygraphy INT32 comparison can overflow and falsely pass at zero tolerancePosiblemente ocupada @0z5a la tomó hace 16 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 6 días
-
onnx-graphsurgeon: Graph.toposort() hits RecursionError when a graph has a local function and an If/Loop nodePosiblemente ocupada @gabortar la tomó hace 39 días. AbiertoModule:GraphSurgeon
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 6 días
-
Polygraphy parse_meta accepts a shape with a missing bracketPosiblemente ocupada @SashaMIT la tomó hace 51 días. AbiertoModule:Polygraphy
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
Los mantenedores suelen responder en 6 días
Todos los issues de NVIDIA/TensorRT
Issues similares
-
[UBSan] spawn_simulator passes null child_options.args to uv_spawn (execvp nonnull violation)Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 79/100
tenstorrent/tt-umd#4017 ·
Los mantenedores suelen responder en 3 días
-
difficulty:easy func:gui prio:low type:bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
NatronGitHub/Natron#1538 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
hyprwm/hyprsunset#103 · 1 comentario ·
-
Feature request
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
qbittorrent/qBittorrent#25338 ·
Los mantenedores suelen responder en 2 días
-
Needs: Attention Needs: Version Info Platform: Android Platform: tvOS
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
react/react-native#60180 · 3 comentarios ·
Los mantenedores suelen responder en 2 días