openvinotoolkit/openvino

[Bug]: inconsistent inference results between OpenVINO and ONNX

Open

#20 872 ouverte le 6 nov. 2023

Voir sur GitHub
 (23 commentaires) (0 réactions) (1 assigné)C++ (3 229 forks)auto 404
bugcategory: ONNX FEgood first issue

Métriques du dépôt

Stars
 (10 286 stars)
Métriques de merge PR
 (Merge moyen 14j 3h) (305 PRs mergées en 30 j)

Description

OpenVINO Version

openvino-nightly 2023.2.0.dev20231101

Operating System

Ubuntu 18.04 (LTS)

Device used for inference

CPU

Framework

ONNX

Model used

https://github.com/jikechao/onnx_models/blob/main/ReverseSequence.onnx

Issue description

For the Given model, OpenVINO and ONNX gave different inference results.

Step-by-step reproduction

Step 1:

Download the model and put it into the same directory with the following test script.

Step 2:

Run the test script:

import onnxruntime as ort
import openvino as ov
import numpy as np

onnx_model_path = './ReverseSequence.onnx'
session = ort.InferenceSession(onnx_model_path)


input_x = np.random.random([4, 4]).astype(np.float32)
input_seq = np.random.randint(0, 2, [4], dtype=np.int64)
input_data = {"x": input_x, "sequence_lens": input_seq}

output_name = session.get_outputs()[0].name
onnx_output = session.run([output_name], input_data)[0]


ov_model = ov.convert_model(onnx_model_path)

ir_path = f"temp_OVIR.xml"
ov.save_model(ov_model, ir_path, compress_to_fp16=False)
core = ov.Core()
model = core.read_model(ir_path)

compiled_model = core.compile_model(model=model, device_name="CPU")

# show the model structure
# input_key = compiled_model.input(0)
output_key = compiled_model.output(0)
# network_input_shape = input_key.shape

ov_result = compiled_model(input_data)[output_key]

np.testing.assert_allclose(onnx_output, ov_result, atol=1e-3)

Relevant log output

AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0.001

Mismatched elements: 8 / 16 (50%)
Max absolute difference: 0.885355
Max relative difference: 1.
 x: array([[0.046832, 0.807565, 0.      , 0.      ],
       [0.717256, 0.383817, 0.      , 0.      ],
       [0.798318, 0.265362, 0.      , 0.      ],
       [0.88803 , 0.997446, 0.      , 0.      ]], dtype=float32)
 y: array([[0.046832, 0.807565, 0.164403, 0.664873],
       [0.717256, 0.383817, 0.127707, 0.20239 ],
       [0.798318, 0.265362, 0.824921, 0.604113],
       [0.88803 , 0.997446, 0.803242, 0.885355]], dtype=float32)

Process finished with exit code 1

Issue submission checklist

  • I'm reporting an issue. It's not a question.
  • I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
  • There is reproducer code and related data files such as images, videos, models, etc.

Guide contributeur