[MIGraphX EP] Model cache reuses a compiled program for a different input shape

オープン
  • 何か問題がありますか？
#33,992 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
58/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
cpp, python
領域
backend, machine-learning

調査の方向性

Start in migraphx_execution_provider.cc at make_hash and the no_input_shape branch in Compute, as identified in the issue. Check how input tensor dimensions contribute to the cache key on the first run and how that key is used to name the cache file. Reproduce with the supplied symbolic-shape model and compare cached runs against uncached runs; done means each input shape loads or compiles the matching program across sessions.

索引モデルが issue の本文から書いたものです。

説明

ep:MIGraphX
Describe the issue

With migraphx_model_cache_dir set, the MIGraphX EP can load a program that was compiled for a different input shape. For a model with symbolic input dimensions, the EP then returns output with the old shape instead of recompiling. Without a cache directory, the same sequence of runs is correct.

Because the cache is on disk, the wrong program keeps being loaded in later processes until the file is deleted.

With real image models at around 263x910, returning to an earlier shape within a session also failed on one model with HIP failure 700: an illegal memory access was encountered.

Expected: each distinct input shape gets its own cached program, or a recompile.

Cause (line numbers from migraphx_execution_provider.cc at d93f5446):

  1. make_hash (line 1297) passes v.size() as MurmurHash3's length, which is a byte count. For the std::vector<std::int64_t> of shape values, only the first few bytes are hashed, so height and width don't affect the result.
  2. On the first Compute of a model with symbolic dimensions, the no_input_shape branch (line 1446) doesn't fill input_shapes; only the else branch does (line 1485). The cache file name at line 1497 is then built from an empty list, so every first run of that model gets the same name.

Fixing make_hash alone doesn't resolve the cross-session case, because of the second problem.

Possible fix: pass v.size() * sizeof(*v.data()) in make_hash, and populate input_shapes from the actual input tensors in the no_input_shape branch before the cache file name is built.

Related: ROCm/onnxruntime#237, ROCm/onnxruntime#245 and ROCm/onnxruntime#246 describe the make_hash part against AMD's fork.

To reproduce
import tempfile

import numpy as np
import onnx
import onnxruntime as ort
from onnx import TensorProto, helper

graph = helper.make_graph(
    [helper.make_node("Mul", ["x", "two"], ["y"])],
    "g",
    [helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 1, "h", "w"])],
    [helper.make_tensor_value_info("y", TensorProto.FLOAT, [1, 1, "h", "w"])],
    [helper.make_tensor("two", TensorProto.FLOAT, [], [2.0])],
)
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 17)])
model.ir_version = 10
model_path = tempfile.mkdtemp() + "/m.onnx"
onnx.save(model, model_path)


def session(cache_dir=None):
    options = {"migraphx_model_cache_dir": cache_dir} if cache_dir else {}
    return ort.InferenceSession(model_path, providers=[("MIGraphXExecutionProvider", options)])


def run(s, h, w):
    return s.run(None, {"x": np.ones([1, 1, h, w], dtype=np.float32)})[0].shape


print("no cache, new sessions:   ", [run(session(), h, w) for h, w in ((4, 8), (6, 8), (4, 10))])

cache = tempfile.mkdtemp()
print("shared cache, new sessions:", [run(session(cache), h, w) for h, w in ((4, 8), (6, 8), (4, 10))])

cache = tempfile.mkdtemp()
s = session(cache)
print("shared cache, one session: ", [run(s, h, w) for h, w in ((4, 8), (6, 8), (4, 8))])

Output:

no cache, new sessions:    [(1, 1, 4, 8), (1, 1, 6, 8), (1, 1, 4, 10)]
shared cache, new sessions: [(1, 1, 4, 8), (1, 1, 4, 8), (1, 1, 4, 8)]
shared cache, one session:  [(1, 1, 4, 8), (1, 1, 6, 8), (1, 1, 6, 8)]
Urgency

Not blocking me: I can work around it by using a separate cache directory per input shape. It does produce wrong output with no error for anyone using one cache directory with more than one input size.

System information
  • Platform: Linux
  • OS Version: Gentoo Linux
  • ONNX Runtime Installation: Other / Unknown (the onnxruntime-migraphx 1.27.1 wheel on PyPI, a community build of this repository's source)
  • ONNX Runtime Version or Commit ID: 1.27.1 (the code described above is unchanged at d93f5446)
  • ONNX Runtime API: Python
  • Architecture: X64
  • Execution Provider: MIGraphX
  • Execution Provider Library Version: ROCm 7.2.0, MIGraphX 2.15 built from source (Radeon VII, gfx906)
主要言語
C++
スター
22.2k
フォーク
4.4k
平均マージ
3日 14時間
マージ済み PR（30日）
278

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

microsoft/onnxruntime のほかの issue

microsoft/onnxruntime の issue をすべて見る

似ている issue

C++ の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。