Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Issue: Error Loading 8 bit quntized ONNX Model on inference- Protobuf Parsing Failed

Open
  • Is something wrong?
#453 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
machine-learning

Research direction

Start with the provided ORTModelForCausalLM.from_pretrained example and the reported protobuf parsing error. Reproduce the failure with the dynamically quantized model and inspect the model-loading path; done means identifying a confirmed cause and documenting or validating a resolution. No repository file or test is named in the issue.

Written by the indexing model from the issue text.

Description

Description

I am facing an error when attempting to load a quantized ONNX model using the ORTModelForCausalLM class from the optimum.onnxruntime library. The error message states: "Failed to load model because protobuf parsing failed."
Context

I am using dynamic quantization for 8-bit quantization on the model before loading it for inference with ONNX.
protobuf_decoding from optimum.onnxruntime import ORTModelForCausalLM
from transformers import pipeline, LlamaTokenizer
import torch

onnx_path = "./llmonnx/"
opt_model = ORTModelForCausalLM.from_pretrained(onnx_path, file_name="model.onnx").to('cuda')
tokenizer = LlamaTokenizer.from_pretrained(onnx_path)
opt_optimum_generator = pipeline("text-generation", model=opt_model, tokenizer=tokenizer, device='cuda')

prompt = "give me translation for this?"
generated_text = opt_optimum_generator(prompt, max_length=512, num_return_sequences=1, truncation=True)
print(generated_text[0]['generated_text']) please assist me if there is any resolution for this Thanks!

Dominant language
C++
Stars
1.7k
Forks
414
Avg merge
15h 1m
Merged PRs (30d)
7

Getting set up

We have not checked this project's setup files yet. Start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/onnxruntime-inference-examples

All issues in microsoft/onnxruntime-inference-examples

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.