RapidOCR on CUDA is 4.4x slower than CPU because cudnn_conv_algo_search defaults to EXHAUSTIVE
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 84/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- ai, performance
Research direction
Start with rapidocr/config.yaml and the RapidOcrModel entry point, then trace how EngineConfig.onnxruntime.cuda_ep_cfg is passed to the ONNX Runtime CUDA provider. Use the setup in docs/usage/gpu.md and the supplied reproduction to compare CPU and CUDA recognition timings; done means the CUDA path no longer incurs the reported repeated convolution-search slowdown.
Written by the indexing model from the issue text.
Description
Bug
#4103 made RapidOCR's ONNX Runtime backend use the CUDA execution provider, and
docs/usage/gpu.md now documents how to enable it. On this machine the result is that
OCR gets 4.4x slower than the CPU it replaces.
RapidOCR defaults EngineConfig.onnxruntime.cuda_ep_cfg.cudnn_conv_algo_search to
EXHAUSTIVE (rapidocr/config.yaml), which re-searches the convolution algorithms every
time an input shape changes. Detection sees one shape per page size and amortizes that
search. Recognition feeds one text-line crop at a time, so its shape changes on nearly
every call and the search never pays for itself — it costs far more than the inference.
Measured on 1c2b794b with nothing patched, an L4 with onnxruntime-gpu 1.29.0 and
rapidocr 3.9.2, one 1224x1584 page with lang=["japan"] (35 text lines), median of three
runs after a warm-up (spread within 0.01s):
| det | cls | rec | total | |
|---|---|---|---|---|
| CPU | 0.96s | 0.03s | 1.66s | 2.65s |
| CUDA (main today) | 0.12s | 0.21s | 11.26s | 11.60s |
CUDA + DEFAULT search |
0.22s | 0.16s | 1.30s | 1.68s |
Detection does get 8x faster on the GPU, as expected. Recognition is what inverts the
result.
Passing "EngineConfig.onnxruntime.cuda_ep_cfg.cudnn_conv_algo_search": "DEFAULT"
alongside the keys #4103 added turns the GPU path into a 1.6x speedup over CPU instead of
a 4.4x slowdown. HEURISTIC does not help — it still searches (11.72s); only DEFAULT
skips it.
The trade-off is small and one-sided: DEFAULT costs detection 0.08s a page, because a
fixed algorithm is not the one EXHAUSTIVE would have found, while recognition saves
about 10s. Setting it per model would be better still, but RapidOCR binds one engine
config to all three (cfg.Det.engine_cfg = cfg.EngineConfig[engine_type]), so it cannot
be chosen per model from Docling.
For the record, this is not a per-node fallback to the CPU. With CUDA requested,
session_state reports All nodes placed on [CUDAExecutionProvider]. Number of nodes: 190 for the detection model, with MemcpyTransformer modified: 0.
Steps to reproduce
On a CUDA host with onnxruntime-gpu installed, following the setup in
docs/usage/gpu.md:
from docling.datamodel.accelerator_options import AcceleratorOptions
from docling.datamodel.pipeline_options import RapidOcrOptions
from docling.models.stages.ocr.rapid_ocr_model import RapidOcrModel
model = RapidOcrModel(
enabled=True,
artifacts_path=None,
options=RapidOcrOptions(backend="onnxruntime", lang=["japan"]),
accelerator_options=AcceleratorOptions(device="cuda", num_threads=4),
)
# time model.reader(page_image_array) against the same call with device="cpu"
Docling version
2.125.0 (measured on 1c2b794b)
Python version
3.13.15
- Dominant language
- Python
- Stars
- 67.2k
- Forks
- 4.8k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 84
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from docling-project/docling
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
docling-project/docling#4314 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
docling-project/docling#4299 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
docling-project/docling#4278 · 1 comment ·
-
ReadingOrderModel produces invalid ProvenanceItem charspans when dehyphenating merged text elements Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
docling-project/docling#4217 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
docling-project/docling#4161 ·
All issues in docling-project/docling
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100