After performing static and dynamic quantization to int8, the inference speed became slower rather than faster · SWivid/F5-TTS#1203

(1 Kommentar) (0 Reaktionen) (0 zugewiesene Personen)Python (2.149 Forks)github user discovery

enhancementhelp wanted

Repository-Metriken

Stars: (14.687 Stars)
PR-Merge-Metriken: (PR-Metriken ausstehend)

Beschreibung

Checks

This template is only for usage issues encountered.
I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
I have searched for existing issues, including closed ones, and couldn't find a solution.
I am using English to submit this issue to facilitate community communication.

Environment Details

Ubuntu 22.04.5 LTS Python 3.10.15 torch 2.5.0a0+872d972e41.nv24.8 onnxruntime-gpu 1.23.0 onnx 1.19.0

Steps to Reproduce

1.Create a new Conda environment. 2.Import the F5-TTS project. 3.Export the transformer blocks from F5-TTS to ONNX format. 4.Use onnxruntime.quant_pre_process to infer input shapes and obtain pre_onnx as the input model for quantization. 5.Perform static quantization with the following settings:

Quantized ops: MatMul, Conv
per_channel=True
extra_options={ "ActivationSymmetric": True, "WeightSymmetric": True }
Use the Aishell dataset (speaker S0002) as the calibration set, keeping all other parameters as default. 6.Compare the inference speed before and after quantization — the quantized ONNX model runs slower than the original FP32 model.

✔️ Expected Behavior

No response

❌ Actual Behavior