Scheduled-job skip blames an application operation when another scheduled job holds the lock

Inizia in internal/engine/schedule.go intorno a skip() alla riga 343 e ai controlli dei lock alle righe 356 e 363-364; confronta la gestione dell’application lock con il percorso di errore di schedule.lock e controlla internal/engine/lock.go:96-97 per l’altro detentore del lock. Traccia come il motivo selezionato raggiunge il file di stato, il notifier e ob status. Il lavoro è completato quando una collisione tra job non viene segnalata come un’operazione dell’applicazione, con una copertura per entrambi i casi distinguibili.

What happens

When a scheduled job stands aside because it could not take schedule.lock, it records:

an application operation is taking its lock

That names an application operation — a deploy, a job run, an exec — as the holder. In the common case it is another scheduled job, and no application operation is involved at all.

schedule.lock is taken by both:

• Application operations, but only around the atomic creation of the application lock (internal/engine/lock.go:96-97), so for milliseconds.
• Every scheduled job run, for as long as it needs (internal/engine/schedule.go:362) — an exclusive job holds it for its whole run, a pinned one until its release lease exists.

So the message describes the rarer holder and omits the frequent one. An operator reading it is pointed at their own deploys and exec sessions, when nothing they do to those will change the outcome.

The runner already distinguishes the two cases it can name accurately:

• stand_aside 'another run of this job is still in progress' — the per-job lock (schedule.go:356), accurate.
• skip 'an application operation holds the deploy lock' — the application lock file, tested directly (schedule.go:364), accurate.

It is only the flock on schedule.lock (schedule.go:363) that guesses, and guesses wrong.

skip() writes the reason it is handed and nothing else (schedule.go:343), so the misleading string is what lands in the state file, the notifier and ob status — there is no better-quality record behind it to fall back on.

Observed

A production application with sixteen scheduled jobs, several sharing cron boundaries (* * * * *, */2 * * * *, two firing together on :00):

schedule sync-sec-submissions skipped 5 firings in a row: an application operation is taking its lock ⚠
schedule sync-bls    skipped 3 firings in a row: an application operation is taking its lock ⚠
schedule sync-cftc-cot skipped 3 firings in a row: an application operation is taking its lock ⚠

Ten of the sixteen showed that reason as their last outcome. This sent me looking at the operator's ob exec sessions, which had run in the same window and were the obvious suspect given the wording. They were not the cause: an exec holds schedule.lock only during application-lock creation, and had it been the application lock the reason would have been the other message. The real contention was job against job.

The diagnosis cost is the point — the message is confident and specific, and it is confidently pointing at the wrong thing.

Expected

The reason names what actually held the lock. The runner cannot see the holder directly, but it does not have to guess either: flock failing here means something holds it, and the two candidates are distinguishable at the point of failure — the application lock file's presence and age is already tested a line later.

Roughly: if the application lock exists and is within its TTL, say so; otherwise say another scheduled job holds it. Failing that, a reason that does not assert a cause it has not established would still be an improvement over one that asserts the wrong one.

Why it matters beyond wording

With catch_up: false a skipped firing is lost rather than deferred, so the count in ob status is missed ingestion, not delayed ingestion. An operator who believes the cause is their own deploy cadence will change something that cannot help, while the real cause — schedule density — goes unexamined.

This is adjacent to #133 but not the same: that is about deploys proceeding during long jobs, this is about a job-versus-job collision being reported as an application operation.