Recognize CUDA backend slower than TensorFlow.js;

Start by mapping the reported processing path across Task Processing, AppAPI, HaRP, the backend, and database writes; the issue does not identify specific files or tests. Compare end-to-end throughput with the previous TensorFlow.js path and investigate the reported stuck jobs, restart recovery, multi-user queue behavior, and ImageNet results. Done means identifying and addressing the relevant causes, with evidence that processing recovers and delivers useful results at improved throughput.

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Steps to reproduce

1. Run Nextcloud AIO with Recognize 13.1.0 and the Recognize Backend ExApp using CUDA on an NVIDIA RTX 3090 Ti.
2. Enable:
   • Face recognition
   • Task Processing / Recognize Backend
   • ImageNet classification
3. Reset face recognition and allow Recognize to rebuild a large photo library containing files belonging to multiple Nextcloud users.
4. Allow the backend to process continuously for several hours.
5. Observe:
   • Task Processing jobs
   • Recognize backend logs
   • CPU and GPU utilization
   • time to first visible People results in Memories
   • generated ImageNet tags
6. Restart the host or otherwise interrupt processing and observe whether queued/running tasks recover automatically.
7. Compare the result with the previous local TensorFlow.js CPU implementation on the same hardware and photo library.
8. Optionally disable ImageNet classification and observe the effect on face-processing throughput.

Expected behaviour

I expected the CUDA backend on an RTX 3090 Ti to provide substantially better end-to-end throughput than the previous local TensorFlow.js CPU implementation.

Specifically:

• Face recognition should become usable significantly faster than CPU-only TensorFlow.js.
• GPU utilization should represent a meaningful portion of the workload.
• Task Processing should recover automatically after a reboot/backend interruption.
• Stale RUNNING jobs should be reclaimed automatically.
• ImageNet tagging quality should be at least comparable to the previous TensorFlow.js implementation.
• Processing should continue predictably across multiple users.
• A host restart should not require manually restarting HaRP or the Recognize backend.
• The user should have some indication of which workload/user is currently being processed.

Actual behaviour

There are several separate observations.

1. End-to-end CUDA performance is slower than the previous CPU implementation

On the same photo library, the previous local TensorFlow.js implementation running entirely on a Ryzen 9 5950X would normally start producing useful tags and face groups within a relatively short period, often well within a few hours and sometimes much sooner.

With the Recognize Backend / CUDA path, after many hours of processing I initially had no visible People results in Memories.

Observed while the backend was actively processing:

• Recognize backend CPU utilization: approximately 62–65%
• RTX 3090 Ti GPU utilization: generally only 17–35%
• GPU memory usage: approximately 2 GB
• Backend logs explicitly showed the face model loading on CUDA:
  • Loading recognize:image:facerecognition processor (buffalo_l) on cuda

The GPU was clearly being used, but it was nowhere near saturated.

The backend was processing batches of approximately 500 files at a time.

Representative observed face-recognition batch durations included approximately:

• 226 seconds
• 253 seconds
• 235 seconds
• 238 seconds
• 239 seconds
• 227 seconds

So many 500-file face-recognition batches took roughly 3.5–4 minutes.

The practical result was that time-to-first-useful-result in Memories was significantly worse than the previous CPU TensorFlow.js implementation.
My current interpretation is that raw CUDA inference is not the limiting factor. The surrounding pipeline appears to dominate:

file crawl -> Task Processing -> AppAPI -> HaRP -> backend -> decode/preprocessing -> CUDA inference -> result return -> DB writes -> clustering

The GPU accelerates only part of that chain.
On a high-core-count CPU such as the Ryzen 9 5950X, the old local implementation appears to be faster end-to-end.

2. Significant CPU bottleneck despite GPU acceleration

Even with CUDA active, CPU utilization remained approximately 62–65% while GPU utilization was usually only 17–35%.
This suggests the backend is frequently waiting on CPU-side work, file handling, preprocessing, task orchestration, transport, or result handling.
For this workload, the RTX 3090 Ti appears underutilized.

The architecture therefore seems to add significant orchestration/handoff overhead compared with the previous local TensorFlow.js path.

3. Large Task Processing backlog and stale RUNNING jobs

We observed a substantial Recognize Task Processing backlog.
Two classification tasks remained stuck in STATUS_RUNNING:

recognize:image:classification
userId=<user>
STATUS_RUNNING
progress=0.00412

and:

recognize:image:classification
userId=<user>
STATUS_RUNNING
progress=0.00198

These remained in RUNNING state for a long period without apparent progress.
Meanwhile, many additional tasks accumulated in STATUS_SCHEDULED.
There were hundreds of scheduled classification and face-recognition jobs.
The backend container itself could be healthy but idle while these stale task states remained.

4. Processing did not recover cleanly after interruption/reboot

After a host reboot/interruption, Recognize did not automatically resume useful processing.
Backend logs showed HaRP connectivity problems such as:

connect to server error: dial tcp ...8782: connection refused

and:

lookup nextcloud-aio-harp ... no such host

The queue itself survived, but useful processing did not resume until I manually restarted:

1. nextcloud-aio-harp
2. nc_app_recognize_backend

Immediately afterward, a new face-recognition task began processing and the CUDA face model loaded.

From an operational perspective, I would expect stale RUNNING jobs and broken worker sessions to be automatically reclaimed using a timeout, lease, heartbeat, or equivalent recovery mechanism.
I deliberately did not create an external watchdog because I believe this recovery logic belongs in the application/backend itself.

5. Multi-user processing order was confusing

The Nextcloud instance contains multiple users.
I initiated/reset face recognition while logged into one user account.
However, the overwhelming majority of visible Task Processing jobs were for a different user account.
There were some scheduled jobs for the initiating user, including both:

recognize:image:classification
recognize:image:facerecognition

but they appeared to be sitting behind a very large backlog belonging to another user.
At one point the actively running face-recognition task was also for the other user.
If Recognize intentionally processes all users globally/sequentially, this is understandable, but it is not obvious from the UI.
From a user perspective, initiating a reset while logged into one account appears as though that account is being rebuilt.

Some indication of:

• current user being processed
• queued jobs by user
• current workload
• progress / files processed

would make this much easier to understand.

6. ImageNet classification quality was substantially worse

While waiting for face recognition, hundreds of new object/system tags began appearing.
Many of the classifications appeared incorrect or unrelated to the actual images.
The results were materially worse than the tags I previously received from the local TensorFlow.js implementation.
This was not limited to one or two marginal classifications; enough poor tags appeared that I disabled ImageNet entirely:

imagenet.enabled = false

while keeping:

faces.enabled = true

After disabling ImageNet, face recognition began surfacing useful People results more quickly.
This suggests object classification was consuming a meaningful portion of the available pipeline capacity.

It also raises a separate quality question around:

• model accuracy
• confidence thresholds
• label mapping
• or potentially result/file association through the task-processing path

7. Face recognition eventually worked, but much more slowly

Once ImageNet was disabled and the backend/HaRP path was recovered, People finally began appearing in Memories.
So the face-recognition model itself does work.
The issue is throughput and time-to-visible-result.
For me the important metric is not raw model inference speed. It is:

reset faces -> usable People appearing in Memories

By that metric, the previous local TensorFlow.js CPU implementation was materially faster on this system.

8. CUDA errors were observed during one graphics-stack failure

During one period of active Recognize CUDA processing, the backend logged:

CUDA failure 999: unknown error

and:

CUDNN_STATUS_INTERNAL_ERROR

followed by:

CUDA runtime error in recognize:image:facerecognition, switching to CPU permanently

At approximately the same time, Windows lost access to the graphics/display stack:

• local console became inaccessible
• RDP became inaccessible
• Windows itself remained partially alive
• a remote management agent still responded
• nvidia-smi stopped responding normally

Windows recorded multiple:

LiveKernelEvent 193

events.

However, I do not currently believe Recognize Backend is proven to be the root cause of the graphics instability.
After completely disabling/stopping the Recognize CUDA backend, the same graphics failure occurred again.
Therefore, Recognize may have exposed/stressed an existing GPU/driver problem, but it is not sufficient by itself to explain the failures.
I am separately investigating the NVIDIA/Windows graphics stack.

Overall performance conclusion

The CUDA backend clearly works technically:

• CUDA initializes
• the model loads on GPU
• GPU utilization is measurable
• face recognition eventually produces useful results

However, on my system the end-to-end experience is currently worse than the previous local TensorFlow.js CPU implementation:

• slower time-to-first-results
• high CPU utilization despite GPU acceleration
• relatively low GPU utilization
• large Task Processing backlog
• stale RUNNING jobs
• poor restart/recovery behavior
• confusing multi-user queue behavior
• significantly worse ImageNet tagging quality
• additional AppAPI/HaRP/task-processing complexity

The main conclusion from my testing is:

GPU inference performance is not translating into higher overall Recognize throughput.

The bottleneck appears to be the pipeline surrounding the model rather than the RTX 3090 Ti itself.

For a Ryzen 9 5950X system, the previous local TensorFlow.js implementation currently feels faster, simpler, and more predictable.

Areas that may be worth investigating

Some possible areas based on the observed behaviour:

• automatic recovery/requeue of stale RUNNING tasks
• worker heartbeat / task lease expiration
• HaRP/backend startup and reconnect behaviour
• visibility of queue depth and currently processed user
• fair multi-user scheduling
• file-transfer / serialization overhead
• image decode/preprocessing CPU overhead
• database/result-write overhead
• model residency / repeated initialization
• larger or more efficient GPU batching
• overlapping preprocessing, inference, and result handling
• ImageNet model accuracy / label mapping / confidence thresholds
• ability to remove only Recognize-generated ImageNet tags
• end-to-end benchmarking in files/hour rather than inference-only benchmarks

Server configuration

Web server: Apache / Nextcloud AIO

Database: PostgreSQL 18.6

PHP version: Not checked

Nextcloud version: 35.0.1 (35.0.1.1)

Recognize version: 13.1.0

Recognize Backend: recognize_backend/1.0.6

Memories version: 10.1.0

AppAPI: 35.0.0

Host CPU: AMD Ryzen 9 5950X, 16C/32T

Host GPU: NVIDIA GeForce RTX 3090 Ti

Host RAM: 64 GB

Host OS: Windows 11 Pro

Container environment: Docker Desktop / Nextcloud AIO

Recognize 13.1.0
Memories 10.1.0
AppAPI 35.0.0

{
  "tensorflow.cores": "0",
  "imagenet.enabled": "false",
  "musicnn.enabled": "false",
  "movinet.enabled": "false",
  "tensorflow.gpu": "false",
  "landmarks.enabled": "true",
  "faces.enabled": "true",
  "enabled": "yes",
  "taskprocessing.enabled": "false",
  "tensorflow.purejs": "true",
  "clusterFaces.status": "true",
  "faces.status": "true",
  "imagenet.status": "true",
  "landmarks.status": "true",
  "faces.batchSize": "500",
  "imagenet.batchSize": "100",
  "landmarks.batchSize": "100",
  "concurrency.enabled": "true"
}

taskprocessing.enabled = true
imagenet.enabled = true
faces.enabled = true

Loading recognize:image:facerecognition processor (buffalo_l) on cuda

CUDA failure 999: unknown error
GPU=0
hostname=recognize_backend
file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc
line=65
expr=cudaMemcpyAsync(...)

CUDA failure 999: unknown error
GPU=0
hostname=recognize_backend
file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc
line=446
expr=cudaStreamSynchronize(...)

CUDA runtime error in recognize:image:facerecognition, switching to CPU permanently

CUDNN failure 4000: CUDNN_STATUS_INTERNAL_ERROR

recognize:image:classification
userId=<user>
STATUS_RUNNING
progress=0.00412

recognize:image:classification
userId=<user>
STATUS_RUNNING
progress=0.00198

Full output of:
sudo -u www-data php occ config:list system

can be provided if needed.

Browser

Browser name: Chrome

Browser version: Current Chrome

Operating system: Windows 11 Pro

No browser-side JavaScript or network error has been identified as relevant so far.

The observed issues are primarily backend/task-processing/performance related.