RapidOCR on CUDA is 4.4x slower than CPU because cudnn_conv_algo_search defaults to EXHAUSTIVE

Open Beginner friendly
#4,167 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
ai, performance

Research direction

Start with rapidocr/config.yaml and the RapidOcrModel entry point, then trace how EngineConfig.onnxruntime.cuda_ep_cfg is passed to the ONNX Runtime CUDA provider. Use the setup in docs/usage/gpu.md and the supplied reproduction to compare CPU and CUDA recognition timings; done means the CUDA path no longer incurs the reported repeated convolution-search slowdown.

Written by the indexing model from the issue text.

Description

Bug

#4103 made RapidOCR's ONNX Runtime backend use the CUDA execution provider, and
docs/usage/gpu.md now documents how to enable it. On this machine the result is that
OCR gets 4.4x slower than the CPU it replaces.

RapidOCR defaults EngineConfig.onnxruntime.cuda_ep_cfg.cudnn_conv_algo_search to
EXHAUSTIVE (rapidocr/config.yaml), which re-searches the convolution algorithms every
time an input shape changes. Detection sees one shape per page size and amortizes that
search. Recognition feeds one text-line crop at a time, so its shape changes on nearly
every call and the search never pays for itself — it costs far more than the inference.

Measured on 1c2b794b with nothing patched, an L4 with onnxruntime-gpu 1.29.0 and
rapidocr 3.9.2, one 1224x1584 page with lang=["japan"] (35 text lines), median of three
runs after a warm-up (spread within 0.01s):

det cls rec total
CPU 0.96s 0.03s 1.66s 2.65s
CUDA (main today) 0.12s 0.21s 11.26s 11.60s
CUDA + DEFAULT search 0.22s 0.16s 1.30s 1.68s

Detection does get 8x faster on the GPU, as expected. Recognition is what inverts the
result.

Passing "EngineConfig.onnxruntime.cuda_ep_cfg.cudnn_conv_algo_search": "DEFAULT"
alongside the keys #4103 added turns the GPU path into a 1.6x speedup over CPU instead of
a 4.4x slowdown. HEURISTIC does not help — it still searches (11.72s); only DEFAULT
skips it.

The trade-off is small and one-sided: DEFAULT costs detection 0.08s a page, because a
fixed algorithm is not the one EXHAUSTIVE would have found, while recognition saves
about 10s. Setting it per model would be better still, but RapidOCR binds one engine
config to all three (cfg.Det.engine_cfg = cfg.EngineConfig[engine_type]), so it cannot
be chosen per model from Docling.

For the record, this is not a per-node fallback to the CPU. With CUDA requested,
session_state reports All nodes placed on [CUDAExecutionProvider]. Number of nodes: 190 for the detection model, with MemcpyTransformer modified: 0.

Steps to reproduce

On a CUDA host with onnxruntime-gpu installed, following the setup in
docs/usage/gpu.md:

from docling.datamodel.accelerator_options import AcceleratorOptions
from docling.datamodel.pipeline_options import RapidOcrOptions
from docling.models.stages.ocr.rapid_ocr_model import RapidOcrModel

model = RapidOcrModel(
    enabled=True,
    artifacts_path=None,
    options=RapidOcrOptions(backend="onnxruntime", lang=["japan"]),
    accelerator_options=AcceleratorOptions(device="cuda", num_threads=4),
)
# time model.reader(page_image_array) against the same call with device="cpu"
Docling version

2.125.0 (measured on 1c2b794b)

Python version

3.13.15

Dominant language
Python
Stars
67.2k
Forks
4.8k
Avg merge
2d 18h
Merged PRs (30d)
84

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from docling-project/docling

All issues in docling-project/docling

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.