openvinotoolkit/openvino

[Bug]: OpenVINO produced inconsistent inference results with ONNX

Open

#20,879 opened on Nov 6, 2023

View on GitHub
 (15 comments) (0 reactions) (1 assignee)C++ (3,229 forks)auto 404
ONNXbugcategory: ONNX FEgood first issue

Repository metrics

Stars
 (10,286 stars)
PR merge metrics
 (Avg merge 14d 3h) (305 merged PRs in 30d)

Description

OpenVINO Version

openvino-nightly 2023.2.0.dev20231101

Operating System

Ubuntu 18.04 (LTS)

Device used for inference

BATCH

Framework

ONNX

Model used

https://github.com/jikechao/onnx_models/blob/main/MaxPool.onnx

Issue description

The given model produces inconsistent predicted results with ONNXRuntime. This inconsistency exists for both CPU/GPU devices. Due to the randomness of generated test inputs, results also have randomness. If there is no inconsistency reproduced, please run the script once again.

Step-by-step reproduction

Step 1: Download the given model and put it into the same directory with the following script

Step 2: Run the following script:

import onnxruntime as ort
import openvino as ov
import numpy as np

onnx_model_path = './MaxPool.onnx'
session = ort.InferenceSession(onnx_model_path)


input_x = np.random.random([1, 1, 5, 5]).astype(np.float32)
input_data = {"x": input_x}
print(input_data)

output_name = session.get_outputs()[0].name
onnx_output = session.run(None, input_data)


ov_model = ov.convert_model(onnx_model_path)

ir_path = f"temp_OVIR.xml"
ov.save_model(ov_model, ir_path, compress_to_fp16=False)
core = ov.Core()
model = core.read_model(ir_path)

compiled_model = core.compile_model(model=model, device_name="CPU")

# show the model structure
# input_key = compiled_model.input(0)
output_key = compiled_model.outputs
# network_input_shape = input_key.shape

for i, output in enumerate(output_key):
    ov_output = compiled_model(input_data)[output]
    np.testing.assert_allclose(onnx_output[i], ov_output, atol=1e-3)

Relevant log output

AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0.001

Mismatched elements: 2 / 4 (50%)
Max absolute difference: 5
Max relative difference: 2.5
 x: array([[[[ 0, 10],
         [ 7, 12]]]], dtype=int64)
 y: array([[[[ 0, 11],
         [ 2, 12]]]], dtype=int64)

Issue submission checklist

  • I'm reporting an issue. It's not a question.
  • I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
  • There is reproducer code and related data files such as images, videos, models, etc.

Contributor guide