[MIGraphX EP] Model cache reuses a compiled program for a different input shape

Abierto
  • ¿Algo no está bien?
#33,992 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
58/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
cpp, python
Área
backend, machine-learning

Línea de trabajo

Start in migraphx_execution_provider.cc at make_hash and the no_input_shape branch in Compute, as identified in the issue. Check how input tensor dimensions contribute to the cache key on the first run and how that key is used to name the cache file. Reproduce with the supplied symbolic-shape model and compare cached runs against uncached runs; done means each input shape loads or compiles the matching program across sessions.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

ep:MIGraphX
Describe the issue

With migraphx_model_cache_dir set, the MIGraphX EP can load a program that was compiled for a different input shape. For a model with symbolic input dimensions, the EP then returns output with the old shape instead of recompiling. Without a cache directory, the same sequence of runs is correct.

Because the cache is on disk, the wrong program keeps being loaded in later processes until the file is deleted.

With real image models at around 263x910, returning to an earlier shape within a session also failed on one model with HIP failure 700: an illegal memory access was encountered.

Expected: each distinct input shape gets its own cached program, or a recompile.

Cause (line numbers from migraphx_execution_provider.cc at d93f5446):

  1. make_hash (line 1297) passes v.size() as MurmurHash3's length, which is a byte count. For the std::vector<std::int64_t> of shape values, only the first few bytes are hashed, so height and width don't affect the result.
  2. On the first Compute of a model with symbolic dimensions, the no_input_shape branch (line 1446) doesn't fill input_shapes; only the else branch does (line 1485). The cache file name at line 1497 is then built from an empty list, so every first run of that model gets the same name.

Fixing make_hash alone doesn't resolve the cross-session case, because of the second problem.

Possible fix: pass v.size() * sizeof(*v.data()) in make_hash, and populate input_shapes from the actual input tensors in the no_input_shape branch before the cache file name is built.

Related: ROCm/onnxruntime#237, ROCm/onnxruntime#245 and ROCm/onnxruntime#246 describe the make_hash part against AMD's fork.

To reproduce
import tempfile

import numpy as np
import onnx
import onnxruntime as ort
from onnx import TensorProto, helper

graph = helper.make_graph(
    [helper.make_node("Mul", ["x", "two"], ["y"])],
    "g",
    [helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 1, "h", "w"])],
    [helper.make_tensor_value_info("y", TensorProto.FLOAT, [1, 1, "h", "w"])],
    [helper.make_tensor("two", TensorProto.FLOAT, [], [2.0])],
)
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)])
model.ir_version = 10
model_path = tempfile.mkdtemp() + "/m.onnx"
onnx.save(model, model_path)


def session(cache_dir=None):
    options = {"migraphx_model_cache_dir": cache_dir} if cache_dir else {}
    return ort.InferenceSession(model_path, providers=[("MIGraphXExecutionProvider", options)])


def run(s, h, w):
    return s.run(None, {"x": np.ones([1, 1, h, w], dtype=np.float32)})[0].shape


print("no cache, new sessions:   ", [run(session(), h, w) for h, w in ((4, 8), (6, 8), (4, 10))])

cache = tempfile.mkdtemp()
print("shared cache, new sessions:", [run(session(cache), h, w) for h, w in ((4, 8), (6, 8), (4, 10))])

cache = tempfile.mkdtemp()
s = session(cache)
print("shared cache, one session: ", [run(s, h, w) for h, w in ((4, 8), (6, 8), (4, 8))])

Output:

no cache, new sessions:    [(1, 1, 4, 8), (1, 1, 6, 8), (1, 1, 4, 10)]
shared cache, new sessions: [(1, 1, 4, 8), (1, 1, 4, 8), (1, 1, 4, 8)]
shared cache, one session:  [(1, 1, 4, 8), (1, 1, 6, 8), (1, 1, 6, 8)]
Urgency

Not blocking me: I can work around it by using a separate cache directory per input shape. It does produce wrong output with no error for anyone using one cache directory with more than one input size.

System information
  • Platform: Linux
  • OS Version: Gentoo Linux
  • ONNX Runtime Installation: Other / Unknown (the onnxruntime-migraphx 1.27.1 wheel on PyPI, a community build of this repository's source)
  • ONNX Runtime Version or Commit ID: 1.27.1 (the code described above is unchanged at d93f5446)
  • ONNX Runtime API: Python
  • Architecture: X64
  • Execution Provider: MIGraphX
  • Execution Provider Library Version: ROCm 7.2.0, MIGraphX 2.15 built from source (Radeon VII, gfx906)
Lenguaje dominante
C++
Estrellas
22.2k
Forks
4.4k
Merge medio
3 d 14 h
PR fusionados (30 d)
278

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/onnxruntime

Todos los issues de microsoft/onnxruntime

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.