Allow setting a limit for simultaneous face clustering runs: ClusterFacesJob runs for different users currently stack up until the server runs out of memory

Read ClusterFacesJob::run() and the job scheduling paths in TaskResultListener.php and MaintenanceJob.php; the issue names these as the sources of overlapping runs. Check how Nextcloud exposes locks and IJobList::scheduleAfter(), then compare a local lock with a distributed cache lock for multi-server cron. Done means clustering concurrency is configurable and competing jobs defer without blocking cron; the issue provides a local patch and measured single-server results.

Describe the feature you'd like to request

An admin-side limit on how many ClusterFacesJob runs can execute at once, across all users. The default can be super conservative (e.g. one or two).

I'm running Recognize 13.1.0 with recognize_backend 1.0.6 (CUDA), Nextcloud 35.0.2 AIO (in an LXC Container on Proxmox 9.2.21).

All my photos are in one group folder that 39 users can see. Detection runs on an Nvidia T1000 8GB and is fast, but clustering runs in PHP inside cron.php, once per user, and a face detection is stored once for every user with a mount. Here 13,961 detections from 4,506 photos became 544,456 rows, so a batch of new faces can mean up to 39 clustering runs of about 14,000 detections each.

Overnight these runs piled up. Cron starts a new cron.php every 5 minutes, and each one picked up the next user's ClusterFacesJob while the earlier ones were still going. By morning 12 clustering runs were active at 0.4 to 0.7 GB each, the oldest almost 10 hours old, the container was at its 4 GiB memory limit with swap full, and Nextcloud had been dropping out all night. I asked Claude Code to trace it, and this is what it found:

Nothing stops the runs from overlapping:

• ClusterFacesJob is a QueuedJob, and QueuedJob::start() deletes the job's row before run() is called (QueuedJob.php#L26-L33). Core's setAllowParallelRuns(false) wouldn't help either, because JobList::getNext() decides that through hasReservedJob(), which looks for a reserved row of the same class (JobList.php#L203, #L402-L407). A running ClusterFacesJob has no row left to find.
• TaskResultListener queues a ClusterFacesJob for each user that got a new detection, after every face task result (TaskResultListener.php#L243-L246).
• MaintenanceJob queues one for every user returned by getUsersForUnclustered(), every 12 hours (MaintenanceJob.php#L41-L43). A full run doesn't take a user off that list: one user went from 7,883 detections without a cluster to 5,672, so all 39 users are queued again twice a day even when no new photos arrive.

The classifier jobs already have a guard: ClassifierJob is a TimedJob and disallows parallel runs unless concurrency.enabled is set. Clustering has nothing similar.

The out-of-memory half of #1558, and #1577, are about the size of one run. The fix for #1558 (80ad6a15) derives the batch size from memory_limit and is already in 13.1.0, the version running here. #1577 shows that formula still overflows at higher limits, and mentions that two runs for the same user can overlap. This request is about the number of runs. Even when each run fits in memory_limit, a new run starting every 5 minutes beside the ones still going will use up the server's memory sooner or later.

Describe the solution you'd like

Check a lock at the start of ClusterFacesJob::run(). If another clustering run holds it, put the job back with IJobList::scheduleAfter() a few minutes out and return, instead of starting a second run. A setting such as clusterFaces.concurrency could allow more than one for admins with the memory to spare.

I'm running this as a local patch (made by Claude Code) now. It uses flock on a file in the temp directory, because the kernel drops the lock if the process dies, so an out-of-memory kill can't leave it stuck:

$lockPath = sys_get_temp_dir() . '/recognize-cluster-faces.lock';
$lock = fopen($lockPath, 'c');
if ($lock === false || !flock($lock, LOCK_EX | LOCK_NB)) {
	$this->jobList->scheduleAfter(self::class, $this->time->getTime() + 600, $argument);
	// log and return
	return;
}
try {
	// unchanged: batch size, calculateClusters()
} finally {
	flock($lock, LOCK_UN);
	fclose($lock);
}

With it, only one clustering run is active at a time. Nextcloud's own job run records for the first 90 minutes show 16 clustering runs of 2 to 5.5 minutes each, peaking between 0.7 and 1.8 GB, and never two at once. Before, a run could go on for 10 hours with a dozen of them competing for memory and 4 cores. The other 258 starts in that window found the lock taken and deferred in under a second. Because getNext() skips jobs whose last_checked is in the future, the deferred jobs don't spin the cron loop. A file lock only covers one server, so an instance with cron on several servers would need a lock in the distributed cache instead.

Describe alternatives you've considered

1. Make clustering a single TimedJob that works through the users needing clustering one by one, instead of one QueuedJob per user. Then the existing setAllowParallelRuns(false) would work, and the per-user queueing in TaskResultListener and MaintenanceJob could become a flag.
2. Run occ recognize:cluster-faces by hand. It clusters users one after another, but the background jobs keep being queued and start alongside it.
3. Give the server more memory. I raised the container from 4 to 12 GiB to get it back up; with the runs still stacking, memory went from 7.3 to 9.1 GB in four minutes and I had to stop them.
4. A smaller batch, as in #1577. It shrinks each run but doesn't limit how many run at once.

🤖 Generated with Claude Code