ClusterFacesJob batch size grows linearly with memory_limit, but clustering memory grows quadratically: OOM at every limit above ~3.5 GB

Start with lib/BackgroundJobs/ClusterFacesJob.php and inspect how memory_limit determines the batch size, then trace calculateClusters() to account for memory used outside HDBSCAN. Review the recognize:cluster-faces help text and the reported measurements when choosing a safe batch size. Also trace QueuedJob::start() and TaskResultListener to understand how runs for one user can overlap; done means batches fit within the memory limit with headroom and duplicate concurrent runs are prevented.

Which version of recognize are you using?

13.1.0 (task processing mode, with the Recognize Backend ExApp), Nextcloud 35.0.1, PHP 8.4.26

Describe the Bug

Follow-up to #1268 and 80ad6a15fc96e3aaa2e9a1917f1b2d981664278b. That change fixes low memory limits, but the same OOM comes back at higher ones.

ClusterFacesJob derives its batch size linearly from memory_limit:

// lib/BackgroundJobs/ClusterFacesJob.php
$batchSize = (int)($memoryBytes * 5_000 / 120_000_0000);

HDBSCAN memory grows roughly with the square of the batch, as the recognize:cluster-faces help text also says ("Memory usage will grow with O(n²)"). The two curves cross: above a certain limit, the batch the job picks needs more memory than the limit that was used to pick it. Raising memory_limit makes this worse, not better.

Measured peak memory of OCA\Recognize\Clustering\HDBSCAN alone (memory_get_peak_usage(true), random 128-d vectors, minClusterSize = minSamples = 5, predict(0.35, 0.5)):

n	peak
2,500	0.10 GB
5,000	0.38 GB
10,000	1.36 GB
17,895	4.21 GB

17,895 is the batch the job computes for memory_limit=4096M. HDBSCAN alone needs 4.21 GB, before the detection entities that calculateClusters() loads. On real data, a 10,000-detection calculateClusters() call peaked at 3.4 GB (vs 1.36 GB for HDBSCAN alone).

Fitting the table, peak ≈ 1.36 GB × (n / 10,000)². With the job's formula, n ≈ 4,474 × limit_in_GiB, so HDBSCAN alone needs about 0.27 × limit² GB. That exceeds the limit itself above about 3.7 GB, and real runs fail earlier because of the extra overhead.

Result at memory_limit=4096M: every run with a full batch fails with

Allowed memory size of 4294967296 bytes exhausted (tried to allocate 1310720 bytes)
at .../recognize/lib/Clustering/MrdBallTree.php#456

This happened 18 times in ~95 minutes in our log. The failing runs ran 45–75 minutes each before dying (oc_job_runs.duration), and then clusterFaces.status is set to false.

Related: ClusterFacesJob is a QueuedJob, which removes itself from the job list before running (QueuedJob::start()). TaskResultListener re-adds it after every face recognition result. So during a backfill, a new run for the same user can start while the previous one is still running. Each run can use up to memory_limit on its own, so parallel runs multiply the memory use.

Expected Behavior

The batch size derived from memory_limit should fit in that limit, for example by scaling with the square root of the available memory and with headroom for the entities calculateClusters() loads on top of HDBSCAN. A run should not take up to an hour before failing.

Also, a second ClusterFacesJob for the same user should not start while one is running: for example, re-add the job only if one isn't already running, or prevent two runs for the same user at once.

To Reproduce

1. Set memory_limit=4096M (or higher) for cron.
2. Have a user with enough unclustered face detections to fill a batch (> ~18k), for example after a recrawl/backfill.
3. ClusterFacesJob fails with the error above on every run, and the faces are never clustered.

The HDBSCAN numbers can be reproduced without real data:

<?php
require '/var/www/html/custom_apps/recognize/vendor/autoload.php';
use OCA\Recognize\Clustering\HDBSCAN;
use OCA\Recognize\Vendor\Rubix\ML\Datasets\Labeled;
mt_srand(1);
$n = (int)$argv[1]; $s = []; $l = [];
for ($i = 0; $i < $n; $i++) {
	$c = $i % 40; $v = [];
	for ($d = 0; $d < 128; $d++) $v[] = sin($c * 7 + $d) * 0.1 + mt_rand() / mt_getrandmax() * 0.02;
	$s[] = $v; $l[] = $i;
}
$t = microtime(true);
(new HDBSCAN(new Labeled($s, $l, false), 5, 5))->predict(0.35, 0.5);
printf("n=%d %.1fs peak %.2f GB\n", $n, microtime(true) - $t, memory_get_peak_usage(true) / 1073741824);

php -d memory_limit=-1 bench.php 17895

Workaround

Run occ recognize:cluster-faces -b 10000 with a higher CLI memory limit. Side note: enabling the PHP JIT for that CLI run (-d opcache.enable_cli=1 -d opcache.jit=tracing -d opcache.jit_buffer_size=256M) cut real 10,000-detection batches from ~570 s to ~100 s for us. It doesn't change memory use.