Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

In-allocation runs: whole-second mtimes on Lustre products_dir trigger Snakemake clock-skew failures

Abierto
  • ¿Algo no está bien?
#971 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
backend, infrastructure

Línea de trabajo

Start by tracing how sp --in-allocation chooses products_dir and how exp_persist and other persistence rules declare outputs; the issue names no file paths or tests. Compare the two proposed fixes, using #953 as a reference for node-local storage and persistence. Done means a chosen workflow change prevents Lustre mtime clock-skew failures while publishing correct products.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Snakemake rejects outputs written to a filesystem with whole-second mtimes when their inputs are on a filesystem with nanosecond mtimes.
This affects single-allocation runs (sp ... --in-allocation) whose outputs.products_dir is on Lustre while the work directory is on node-local NVMe.

On the Alliance cluster Fir, /scratch is Lustre and stores mtimes to the second (stat shows .000000000), while $SLURM_TMPDIR stores nanoseconds.
When a rule reads an input from the work directory and writes its output into products_dir within the same second, the output's truncated mtime is earlier than the input's.
Snakemake then raises its "older modification time … clock skew" WorkflowError, deletes the output and retries the job.

  • In the first five Fir production batches (25 tiles each, code 61b6255f), this happened 14–55 times per batch, mostly in exp_persist.
  • Retries usually land in a later second, so most batches finish, and their products are correct: the PSF models and exposure maps of 19 exposures processed on both Fir and Nibi are byte-identical.
  • One batch ran out of retries on two exp_persist jobs and failed.
  • On Nibi, /project and /scratch are NFS with nanosecond mtimes, and the error does not occur in any of ~480 batch logs.

Workaround used on Fir: set products_dir on node-local disk for the whole run, then copy the finished tree to shared storage once at the end.

Possible fixes in the workflow:

  1. sp --in-allocation keeps products in the work directory and publishes the finished tree to products_dir as a final step. This is the Fir workaround made the default, and it also gives a single point where the published tree is complete.
  2. The persisting rules (exp_persist and the other rules that copy into products_dir) declare their products outside Snakemake's output set, so the mtime check never compares across filesystems.

Option 1 seems simpler to me, and it matches how #953 already keeps exposure stores node-local and persists only the per-exposure products.
Any Lustre-backed cluster (Fir, Narval, Rorqual, and possibly others) will hit this as soon as production moves there.

— Claude (Opus 5.5) on behalf of Cail

Lenguaje dominante
Python
Estrellas
18
Forks
14
Merge medio
4 d 14 h
PR fusionados (30 d)
44

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de CosmoStat/shapepipe

Todos los issues de CosmoStat/shapepipe

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.