openvinotoolkit/openvino
View on GitHub[Bug]: OpenVINO produced inconsistent inference results with ONNX
Open
#20,879 opened on Nov 6, 2023
ONNXbugcategory: ONNX FEgood first issue
Repository metrics
- Stars
- (10,286 stars)
- PR merge metrics
- (Avg merge 14d 3h) (305 merged PRs in 30d)
Description
OpenVINO Version
openvino-nightly 2023.2.0.dev20231101
Operating System
Ubuntu 18.04 (LTS)
Device used for inference
BATCH
Framework
ONNX
Model used
https://github.com/jikechao/onnx_models/blob/main/MaxPool.onnx
Issue description
The given model produces inconsistent predicted results with ONNXRuntime. This inconsistency exists for both CPU/GPU devices. Due to the randomness of generated test inputs, results also have randomness. If there is no inconsistency reproduced, please run the script once again.
Step-by-step reproduction
Step 1: Download the given model and put it into the same directory with the following script
Step 2: Run the following script:
import onnxruntime as ort
import openvino as ov
import numpy as np
onnx_model_path = './MaxPool.onnx'
session = ort.InferenceSession(onnx_model_path)
input_x = np.random.random([1, 1, 5, 5]).astype(np.float32)
input_data = {"x": input_x}
print(input_data)
output_name = session.get_outputs()[0].name
onnx_output = session.run(None, input_data)
ov_model = ov.convert_model(onnx_model_path)
ir_path = f"temp_OVIR.xml"
ov.save_model(ov_model, ir_path, compress_to_fp16=False)
core = ov.Core()
model = core.read_model(ir_path)
compiled_model = core.compile_model(model=model, device_name="CPU")
# show the model structure
# input_key = compiled_model.input(0)
output_key = compiled_model.outputs
# network_input_shape = input_key.shape
for i, output in enumerate(output_key):
ov_output = compiled_model(input_data)[output]
np.testing.assert_allclose(onnx_output[i], ov_output, atol=1e-3)
Relevant log output
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0.001
Mismatched elements: 2 / 4 (50%)
Max absolute difference: 5
Max relative difference: 2.5
x: array([[[[ 0, 10],
[ 7, 12]]]], dtype=int64)
y: array([[[[ 0, 11],
[ 2, 12]]]], dtype=int64)
Issue submission checklist
- I'm reporting an issue. It's not a question.
- I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
- There is reproducer code and related data files such as images, videos, models, etc.