Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

In-allocation runs: whole-second mtimes on Lustre products_dir trigger Snakemake clock-skew failures

Offen
  • Stimmt etwas nicht?
#971 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
45/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
backend, infrastructure

Rechercherichtung

Start by tracing how sp --in-allocation chooses products_dir and how exp_persist and other persistence rules declare outputs; the issue names no file paths or tests. Compare the two proposed fixes, using #953 as a reference for node-local storage and persistence. Done means a chosen workflow change prevents Lustre mtime clock-skew failures while publishing correct products.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Snakemake rejects outputs written to a filesystem with whole-second mtimes when their inputs are on a filesystem with nanosecond mtimes.
This affects single-allocation runs (sp ... --in-allocation) whose outputs.products_dir is on Lustre while the work directory is on node-local NVMe.

On the Alliance cluster Fir, /scratch is Lustre and stores mtimes to the second (stat shows .000000000), while $SLURM_TMPDIR stores nanoseconds.
When a rule reads an input from the work directory and writes its output into products_dir within the same second, the output's truncated mtime is earlier than the input's.
Snakemake then raises its "older modification time … clock skew" WorkflowError, deletes the output and retries the job.

  • In the first five Fir production batches (25 tiles each, code 61b6255f), this happened 14–55 times per batch, mostly in exp_persist.
  • Retries usually land in a later second, so most batches finish, and their products are correct: the PSF models and exposure maps of 19 exposures processed on both Fir and Nibi are byte-identical.
  • One batch ran out of retries on two exp_persist jobs and failed.
  • On Nibi, /project and /scratch are NFS with nanosecond mtimes, and the error does not occur in any of ~480 batch logs.

Workaround used on Fir: set products_dir on node-local disk for the whole run, then copy the finished tree to shared storage once at the end.

Possible fixes in the workflow:

  1. sp --in-allocation keeps products in the work directory and publishes the finished tree to products_dir as a final step. This is the Fir workaround made the default, and it also gives a single point where the published tree is complete.
  2. The persisting rules (exp_persist and the other rules that copy into products_dir) declare their products outside Snakemake's output set, so the mtime check never compares across filesystems.

Option 1 seems simpler to me, and it matches how #953 already keeps exposure stores node-local and persists only the per-exposure products.
Any Lustre-backed cluster (Fir, Narval, Rorqual, and possibly others) will hit this as soon as production moves there.

— Claude (Opus 5.5) on behalf of Cail

Vorherrschende Sprache
Python
Sterne
18
Forks
14
Ø Merge
4 T. 14 Std.
Gemergte PRs (30 T.)
44

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus CosmoStat/shapepipe

Alle Issues in CosmoStat/shapepipe

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.