Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

ClusterFacesJob batch size grows linearly with memory_limit, but clustering memory grows quadratically: OOM at every limit above ~3.5 GB

Aperta
  • Qualcosa non va?
#1,577 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
machine-learning, php
Ambito
backend, performance

Direzione di ricerca

Start with lib/BackgroundJobs/ClusterFacesJob.php and inspect how memory_limit determines the batch size, then trace calculateClusters() to account for memory used outside HDBSCAN. Review the recognize:cluster-faces help text and the reported measurements when choosing a safe batch size. Also trace QueuedJob::start() and TaskResultListener to understand how runs for one user can overlap; done means batches fit within the memory limit with headroom and duplicate concurrent runs are prevented.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Which version of recognize are you using?

13.1.0 (task processing mode, with the Recognize Backend ExApp), Nextcloud 35.0.1, PHP 8.4.26

Describe the Bug

Follow-up to #1268 and 80ad6a15fc96e3aaa2e9a1917f1b2d981664278b. That change fixes low memory limits, but the same OOM comes back at higher ones.

ClusterFacesJob derives its batch size linearly from memory_limit:

// lib/BackgroundJobs/ClusterFacesJob.php
$batchSize = (int)($memoryBytes * 5_000 / 120_000_0000);

HDBSCAN memory grows roughly with the square of the batch, as the recognize:cluster-faces help text also says ("Memory usage will grow with O(n²)"). The two curves cross: above a certain limit, the batch the job picks needs more memory than the limit that was used to pick it. Raising memory_limit makes this worse, not better.

Measured peak memory of OCA\Recognize\Clustering\HDBSCAN alone (memory_get_peak_usage(true), random 128-d vectors, minClusterSize = minSamples = 5, predict(0.35, 0.5)):

n peak
2,500 0.10 GB
5,000 0.38 GB
10,000 1.36 GB
17,895 4.21 GB

17,895 is the batch the job computes for memory_limit=4096M. HDBSCAN alone needs 4.21 GB, before the detection entities that calculateClusters() loads. On real data, a 10,000-detection calculateClusters() call peaked at 3.4 GB (vs 1.36 GB for HDBSCAN alone).

Fitting the table, peak ≈ 1.36 GB × (n / 10,000)². With the job's formula, n ≈ 4,474 × limit_in_GiB, so HDBSCAN alone needs about 0.27 × limit² GB. That exceeds the limit itself above about 3.7 GB, and real runs fail earlier because of the extra overhead.

Result at memory_limit=4096M: every run with a full batch fails with

Allowed memory size of 4294967296 bytes exhausted (tried to allocate 1310720 bytes)
at .../recognize/lib/Clustering/MrdBallTree.php#456

This happened 18 times in ~95 minutes in our log. The failing runs ran 45–75 minutes each before dying (oc_job_runs.duration), and then clusterFaces.status is set to false.

Related: ClusterFacesJob is a QueuedJob, which removes itself from the job list before running (QueuedJob::start()). TaskResultListener re-adds it after every face recognition result. So during a backfill, a new run for the same user can start while the previous one is still running. Each run can use up to memory_limit on its own, so parallel runs multiply the memory use.

Expected Behavior

The batch size derived from memory_limit should fit in that limit, for example by scaling with the square root of the available memory and with headroom for the entities calculateClusters() loads on top of HDBSCAN. A run should not take up to an hour before failing.

Also, a second ClusterFacesJob for the same user should not start while one is running: for example, re-add the job only if one isn't already running, or prevent two runs for the same user at once.

To Reproduce
  1. Set memory_limit=4096M (or higher) for cron.
  2. Have a user with enough unclustered face detections to fill a batch (> ~18k), for example after a recrawl/backfill.
  3. ClusterFacesJob fails with the error above on every run, and the faces are never clustered.

The HDBSCAN numbers can be reproduced without real data:

<?php
require '/var/www/html/custom_apps/recognize/vendor/autoload.php';
use OCA\Recognize\Clustering\HDBSCAN;
use OCA\Recognize\Vendor\Rubix\ML\Datasets\Labeled;
mt_srand(1);
$n = (int)$argv[1]; $s = []; $l = [];
for ($i = 0; $i < $n; $i++) {
	$c = $i % 40; $v = [];
	for ($d = 0; $d < 128; $d++) $v[] = sin($c * 7 + $d) * 0.1 + mt_rand() / mt_getrandmax() * 0.02;
	$s[] = $v; $l[] = $i;
}
$t = microtime(true);
(new HDBSCAN(new Labeled($s, $l, false), 5, 5))->predict(0.35, 0.5);
printf("n=%d %.1fs peak %.2f GB\n", $n, microtime(true) - $t, memory_get_peak_usage(true) / 1073741824);

php -d memory_limit=-1 bench.php 17895

Workaround

Run occ recognize:cluster-faces -b 10000 with a higher CLI memory limit. Side note: enabling the PHP JIT for that CLI run (-d opcache.enable_cli=1 -d opcache.jit=tracing -d opcache.jit_buffer_size=256M) cut real 10,000-detection batches from ~570 s to ~100 s for us. It doesn't change memory use.

Lingua principale
PHP
Stelle
701
Fork
68
Merge medio
10g 21h
PR unite (30g)
2

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di nextcloud/recognize

Tutte le issue di nextcloud/recognize

Issue simili

Altre issue su PHP

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.