Try to convert the original phi-3-mini-4k and put it in the provided APP, but get the exception.

Start with the two conversion command sequences and the Android stack trace, then inspect MainActivity.java at lines 48, 51, 163, and 185 and GenAIWrapper.java:22. Compare the generated model files with the provided model, especially model.onnx.data; done means the replacement model loads successfully in the Android app.

According to the ONNX model provided in the example (microsoft/Phi-3-mini-4k-instruct-onnx/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4), following the quantization parameters as described in the folder name (cpu, int4, rtn, block-32, acc-level-4), I attempted to convert the original phi-3-mini-4k model. The model was generated successfully. However, it fails when replacing the original model, regardless of the two conversion methods I used as follows. I found the size of .onnx is different from the provioded version. Are there any suggestions to convert the onnx model to match the official version?

**Method 1:

convert to ONNX

!/data/miniconda/envs/onnx_env/bin/optimum-cli export onnx
--model /data/www/llm_api/dist/{model}/
--task text-generation-with-past ./{model}

quantization

!bash run_quant.sh --model_input=./{model}
--model_output=./{model}/quantized/model
--algorithm=RTN --block_size=32 --accuracy_level=4

**Method 2:

build ONNX

!/data/miniconda/envs/onnx_env/bin/python3 -m onnxruntime_genai.models.builder
-m /data/www/llm_api/dist/{model}/
-o ./{model}/
-p int4
-e cpu
--extra_options int4_block_size=32
--extra_options int4_accuracy_level=4 \

The exception message in Android is as attached.

FATAL EXCEPTION: main
Process: ai.onnxruntime.genai.demo, PID: 29596
java.lang.RuntimeException: Unable to start activity ComponentInfo{ai.onnxruntime.genai.demo/ai.onnxruntime.genai.demo.MainActivity}: java.lang.RuntimeException: ai.onnxruntime.genai.demo.GenAIException: Deserialize tensor model.layers.21.mlp.gate_proj.MatMul.weight_Q4 failed.GetFileLength for /data/user/0/ai.onnxruntime.genai.demo/files/model.onnx.data failed:Invalid fd was supplied: -1
  at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3742)
  at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3879)
  at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:108)
  at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
  at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
  at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2373)
  at android.os.Handler.dispatchMessage(Handler.java:106)
  at android.os.Looper.loopOnce(Looper.java:242)
  at android.os.Looper.loop(Looper.java:359)
  at android.app.ActivityThread.main(ActivityThread.java:8114)
  at java.lang.reflect.Method.invoke(Native Method)
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:942)
Caused by: java.lang.RuntimeException: ai.onnxruntime.genai.demo.GenAIException: Deserialize tensor model.layers.21.mlp.gate_proj.MatMul.weight_Q4 failed.GetFileLength for /data/user/0/ai.onnxruntime.genai.demo/files/model.onnx.data failed:Invalid fd was supplied: -1
  at ai.onnxruntime.genai.demo.MainActivity.onCreate(MainActivity.java:51)
  at android.app.Activity.performCreate(Activity.java:8356)
  at android.app.Activity.performCreate(Activity.java:8335)
  at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1385)
  at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3723)
  at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3879)
  at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:108)
  at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
  at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
  at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2373)
  at android.os.Handler.dispatchMessage(Handler.java:106)
  at android.os.Looper.loopOnce(Looper.java:242)
  at android.os.Looper.loop(Looper.java:359)
  at android.app.ActivityThread.main(ActivityThread.java:8114)
  at java.lang.reflect.Method.invoke(Native Method)
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:942)
Caused by: ai.onnxruntime.genai.demo.GenAIException: Deserialize tensor model.layers.21.mlp.gate_proj.MatMul.weight_Q4 failed.GetFileLength for /data/user/0/ai.onnxruntime.genai.demo/files/model.onnx.data failed:Invalid fd was supplied: -1
  at ai.onnxruntime.genai.demo.GenAIWrapper.loadModel(Native Method)
  at ai.onnxruntime.genai.demo.GenAIWrapper.(GenAIWrapper.java:22)
  at ai.onnxruntime.genai.demo.MainActivity.createGenAIWrapper(MainActivity.java:185)
  at ai.onnxruntime.genai.demo.MainActivity.downloadModels(MainActivity.java:163)
  at ai.onnxruntime.genai.demo.MainActivity.onCreate(MainActivity.java:48)
  at android.app.Activity.performCreate(Activity.java:8356)
  at android.app.Activity.performCreate(Activity.java:8335)
  at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1385)
  at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3723)
  at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3879)
  at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:108)
  at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
  at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
  at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2373)
  at android.os.Handler.dispatchMessage(Handler.java:106)
  at android.os.Looper.loopOnce(Looper.java:242)
  at android.os.Looper.loop(Looper.java:359)
  at android.app.ActivityThread.main(ActivityThread.java:8114)
  at java.lang.reflect.Method.invoke(Native Method)
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:942)