Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

ClusterFacesJob batch size grows linearly with memory_limit, but clustering memory grows quadratically: OOM at every limit above ~3.5 GB

オープン
  • 何か問題がありますか？
#1,577 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
machine-learning, php
領域
backend, performance

調査の方向性

Start with lib/BackgroundJobs/ClusterFacesJob.php and inspect how memory_limit determines the batch size, then trace calculateClusters() to account for memory used outside HDBSCAN. Review the recognize:cluster-faces help text and the reported measurements when choosing a safe batch size. Also trace QueuedJob::start() and TaskResultListener to understand how runs for one user can overlap; done means batches fit within the memory limit with headroom and duplicate concurrent runs are prevented.

索引モデルが issue の本文から書いたものです。

説明

Which version of recognize are you using?

13.1.0 (task processing mode, with the Recognize Backend ExApp), Nextcloud 35.0.1, PHP 8.4.26

Describe the Bug

Follow-up to #1268 and 80ad6a15fc96e3aaa2e9a1917f1b2d981664278b. That change fixes low memory limits, but the same OOM comes back at higher ones.

ClusterFacesJob derives its batch size linearly from memory_limit:

// lib/BackgroundJobs/ClusterFacesJob.php
$batchSize = (int)($memoryBytes * 5_000 / 120_000_0000);

HDBSCAN memory grows roughly with the square of the batch, as the recognize:cluster-faces help text also says ("Memory usage will grow with O(n²)"). The two curves cross: above a certain limit, the batch the job picks needs more memory than the limit that was used to pick it. Raising memory_limit makes this worse, not better.

Measured peak memory of OCA\Recognize\Clustering\HDBSCAN alone (memory_get_peak_usage(true), random 128-d vectors, minClusterSize = minSamples = 5, predict(0.35, 0.5)):

n peak
2,500 0.10 GB
5,000 0.38 GB
10,000 1.36 GB
17,895 4.21 GB

17,895 is the batch the job computes for memory_limit=4096M. HDBSCAN alone needs 4.21 GB, before the detection entities that calculateClusters() loads. On real data, a 10,000-detection calculateClusters() call peaked at 3.4 GB (vs 1.36 GB for HDBSCAN alone).

Fitting the table, peak ≈ 1.36 GB × (n / 10,000)². With the job's formula, n ≈ 4,474 × limit_in_GiB, so HDBSCAN alone needs about 0.27 × limit² GB. That exceeds the limit itself above about 3.7 GB, and real runs fail earlier because of the extra overhead.

Result at memory_limit=4096M: every run with a full batch fails with

Allowed memory size of 4294967296 bytes exhausted (tried to allocate 1310720 bytes)
at .../recognize/lib/Clustering/MrdBallTree.php#456

This happened 18 times in ~95 minutes in our log. The failing runs ran 45–75 minutes each before dying (oc_job_runs.duration), and then clusterFaces.status is set to false.

Related: ClusterFacesJob is a QueuedJob, which removes itself from the job list before running (QueuedJob::start()). TaskResultListener re-adds it after every face recognition result. So during a backfill, a new run for the same user can start while the previous one is still running. Each run can use up to memory_limit on its own, so parallel runs multiply the memory use.

Expected Behavior

The batch size derived from memory_limit should fit in that limit, for example by scaling with the square root of the available memory and with headroom for the entities calculateClusters() loads on top of HDBSCAN. A run should not take up to an hour before failing.

Also, a second ClusterFacesJob for the same user should not start while one is running: for example, re-add the job only if one isn't already running, or prevent two runs for the same user at once.

To Reproduce
  1. Set memory_limit=4096M (or higher) for cron.
  2. Have a user with enough unclustered face detections to fill a batch (> ~18k), for example after a recrawl/backfill.
  3. ClusterFacesJob fails with the error above on every run, and the faces are never clustered.

The HDBSCAN numbers can be reproduced without real data:

<?php
require '/var/www/html/custom_apps/recognize/vendor/autoload.php';
use OCA\Recognize\Clustering\HDBSCAN;
use OCA\Recognize\Vendor\Rubix\ML\Datasets\Labeled;
mt_srand(1);
$n = (int)$argv[1]; $s = []; $l = [];
for ($i = 0; $i < $n; $i++) {
	$c = $i % 40; $v = [];
	for ($d = 0; $d < 128; $d++) $v[] = sin($c * 7 + $d) * 0.1 + mt_rand() / mt_getrandmax() * 0.02;
	$s[] = $v; $l[] = $i;
}
$t = microtime(true);
(new HDBSCAN(new Labeled($s, $l, false), 5, 5))->predict(0.35, 0.5);
printf("n=%d %.1fs peak %.2f GB\n", $n, microtime(true) - $t, memory_get_peak_usage(true) / 1073741824);

php -d memory_limit=-1 bench.php 17895

Workaround

Run occ recognize:cluster-faces -b 10000 with a higher CLI memory limit. Side note: enabling the PHP JIT for that CLI run (-d opcache.enable_cli=1 -d opcache.jit=tracing -d opcache.jit_buffer_size=256M) cut real 10,000-detection batches from ~570 s to ~100 s for us. It doesn't change memory use.

主要言語
PHP
スター
701
フォーク
68
平均マージ
18時間 9分
マージ済み PR（30日）
1

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

nextcloud/recognize のほかの issue

nextcloud/recognize の issue をすべて見る

似ている issue

PHP の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。