Bug Report: Recognize Backend 1.0.5 — CUDA Initialization Failure Due to Missing "libcublasLt.so.11"

Start by reviewing the backend Docker image build configuration and how it installs ONNX Runtime GPU and CUDA libraries. Verify the linked dependencies of libonnxruntime_providers_cuda.so against the CUDA runtime packaged in version 1.0.5. Done means identifying the incompatibility and providing a compatible image or supported upgrade path that restores GPU inference.

Summary

The official Nextcloud Recognize backend Docker image ("ghcr.io/nextcloud/recognize_backend:1.0.5") is producing repeated CUDA initialization errors because the bundled ONNX Runtime GPU provider attempts to load a CUDA 11 library ("libcublasLt.so.11"), while the container provides CUDA 12 libraries.

The container starts successfully, reports a healthy status, and continues processing image classification and face recognition tasks. However, the CUDA execution provider fails to initialize, suggesting that inference is falling back to CPU execution.

This appears to be a dependency compatibility issue within the distributed backend image rather than a host GPU hardware issue.

Environment

Component| Configuration
Host operating system| Windows 11 Pro 26H2
Container runtime| Docker
GPU| NVIDIA GeForce RTX 3090 Ti
Application| Nextcloud Recognize with external backend (ExApp)
Backend image| "ghcr.io/nextcloud/recognize_backend:1.0.5"
Image digest (local image ID)| "sha256:4f1c1e177d22b74d079b67032e0c402e18e19ee3dd7a745a193180dccf2b6e3f"
Installed ONNX Runtime GPU| "onnxruntime-gpu 1.18.1"
CUDA toolkit in container| CUDA 12.2
Available cuBLAS libraries| "libcublasLt.so.12", "libcublasLt.so.13"
Missing dependency| "libcublasLt.so.11"

The backend was deployed using the official developer-supplied container image, without manually replacing its ONNX Runtime or CUDA libraries.

Observed behavior

On October 19, 2026, at approximately 17:59 UTC (13:59 EDT), the backend logged five consecutive CUDA provider initialization failures.

Representative log excerpt:

[E:onnxruntime:Default, provider_bridge_ort.cc:1745
TryGetProviderInfo_CUDA]

/onnxruntime_src/onnxruntime/core/session/
provider_bridge_ort.cc:1426

onnxruntime::Provider&
onnxruntime::ProviderLibrary::Get()

[ONNXRuntimeError] : 1 : FAIL :

Failed to load library
libonnxruntime_providers_cuda.so

with error:

libcublasLt.so.11:
cannot open shared object file:
No such file or directory

The error repeated five times within approximately one second.

A subsequent log review at approximately 21:07 EDT found no additional occurrences of this specific CUDA error after the initial five failures.

However, the absence of repeated errors does not establish that GPU acceleration has recovered, as initialization may occur only when relevant models or worker processes start.

Dependency verification

The following checks were executed directly inside the running container.

ONNX Runtime package:

Name: onnxruntime-gpu
Version: 1.18.1
Location:
/usr/local/lib/python3.10/dist-packages

CUDA libraries identified:

/usr/local/cuda-12.2/targets/x86_64-linux/lib/
libcublasLt.so
libcublasLt.so.12
libcublasLt.so.12.2.5.6

/usr/local/lib/python3.10/dist-packages/nvidia/cu13/lib/
libcublasLt.so.13

ONNX Runtime CUDA provider:

/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/
libonnxruntime_providers_cuda.so

A filesystem search did not identify "libcublasLt.so.11" in the container.

This indicates that the installed CUDA provider binary is attempting to link against a library version that is not packaged in the running image.

Actual versus expected behavior

Expected:

The officially distributed CUDA-enabled Recognize backend should initialize the ONNX Runtime CUDA execution provider successfully when deployed on a compatible NVIDIA GPU host.

Image classification and face recognition should be able to use GPU inference without manual dependency adjustments.

Actual:

• The backend starts and remains healthy.
• Nextcloud successfully submits recognition tasks.
• Classification and face recognition endpoints return HTTP 200 responses.
• The CUDA provider fails to load because of a missing shared library.
• Tasks continue executing, but actual GPU execution versus CPU fallback has not been independently verified.

Additional observations

The backend is actively processing tasks and has not experienced an observable container outage.

Separate log entries include image-processing errors relating to truncated source image files and occasional HTTP/3 downgrade warnings. These appear distinct from the CUDA library problem and are not the subject of this report.

The principal concern is loss of GPU acceleration despite an otherwise operational backend.

Suspected cause

The image appears to bundle an ONNX Runtime GPU provider built against CUDA 11, while its installed CUDA runtime includes CUDA 12.2.

ONNX Runtime 1.18.1 exists in CUDA-specific build variants, so specifying "onnxruntime-gpu==1.18.1" alone may not ensure the expected CUDA compatibility.

Possible explanations include:

1. The Docker build installs a CUDA 11-compatible ONNX Runtime wheel into a CUDA 12 image.
2. A dependency resolution or build-cache issue selects an unintended ONNX Runtime wheel.
3. The distributed image contains runtime libraries that do not match those required by the compiled ONNX Runtime provider.

These are hypotheses; the actual image build configuration needs developer verification.

Requested investigation

Could the maintainers please:

1. Confirm the intended CUDA version for the "recognize_backend:1.0.5" image and the exact ONNX Runtime GPU wheel used during the build.
2. Verify the linked dependencies of "libonnxruntime_providers_cuda.so", particularly whether it requires "libcublasLt.so.11" or ".12".
3. Review image packaging to ensure that the ONNX Runtime provider, CUDA runtime, cuBLAS, and cuDNN versions are mutually compatible.
4. Provide a corrected image or supported upgrade path that restores GPU inference without requiring manual modifications inside the running container.
5. Consider adding a startup GPU-provider validation check so that CUDA initialization failure is clearly surfaced rather than allowing the container to report healthy while GPU acceleration may be unavailable.

Questions for maintainers

• Is CUDA 12.2 the intended runtime for backend version 1.0.5?
• Is the installed "onnxruntime-gpu 1.18.1" wheel expected to require "libcublasLt.so.11"?
• Does the backend intentionally support CPU fallback when GPU initialization fails?
• Is there a newer backend release or recommended image tag with corrected CUDA dependencies?

Current status

The container remains healthy and recognition tasks are processing.

No container modifications, dependency replacements, library symlinks, or other manual workarounds have been applied.

I would prefer to retain the official image and resolve this through an upstream fix rather than introduce an unsupported local patch.

Thank you for investigating.