In-allocation runs: whole-second mtimes on Lustre products_dir trigger Snakemake clock-skew failures
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 45/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- python
调研方向
Start by tracing how sp --in-allocation chooses products_dir and how exp_persist and other persistence rules declare outputs; the issue names no file paths or tests. Compare the two proposed fixes, using #953 as a reference for node-local storage and persistence. Done means a chosen workflow change prevents Lustre mtime clock-skew failures while publishing correct products.
由索引模型根据 Issue 内容生成。
描述
Snakemake rejects outputs written to a filesystem with whole-second mtimes when their inputs are on a filesystem with nanosecond mtimes.
This affects single-allocation runs (sp ... --in-allocation) whose outputs.products_dir is on Lustre while the work directory is on node-local NVMe.
On the Alliance cluster Fir, /scratch is Lustre and stores mtimes to the second (stat shows .000000000), while $SLURM_TMPDIR stores nanoseconds.
When a rule reads an input from the work directory and writes its output into products_dir within the same second, the output's truncated mtime is earlier than the input's.
Snakemake then raises its "older modification time … clock skew" WorkflowError, deletes the output and retries the job.
- In the first five Fir production batches (25 tiles each, code
61b6255f), this happened 14–55 times per batch, mostly inexp_persist. - Retries usually land in a later second, so most batches finish, and their products are correct: the PSF models and exposure maps of 19 exposures processed on both Fir and Nibi are byte-identical.
- One batch ran out of retries on two
exp_persistjobs and failed. - On Nibi,
/projectand/scratchare NFS with nanosecond mtimes, and the error does not occur in any of ~480 batch logs.
Workaround used on Fir: set products_dir on node-local disk for the whole run, then copy the finished tree to shared storage once at the end.
Possible fixes in the workflow:
sp --in-allocationkeeps products in the work directory and publishes the finished tree toproducts_diras a final step. This is the Fir workaround made the default, and it also gives a single point where the published tree is complete.- The persisting rules (
exp_persistand the other rules that copy intoproducts_dir) declare their products outside Snakemake's output set, so the mtime check never compares across filesystems.
Option 1 seems simpler to me, and it matches how #953 already keeps exposure stores node-local and persists only the per-exposure products.
Any Lustre-backed cluster (Fir, Narval, Rorqual, and possibly others) will hit this as soon as production moves there.
— Claude (Opus 5.5) on behalf of Cail
- 主要语言
- Python
- 星标
- 18
- 派生
- 14
- 平均合并
- 4 天 14 小时
- 30 天内合并 PR
- 44
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
CosmoStat/shapepipe 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 74/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 85/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 75/100
维护者通常 1 天内回复
-
难度 5/5 一周以上 新手友好度 20/100
维护者通常 1 天内回复
-
难度 4/5 3-5 天 新手友好度 54/100
维护者通常 1 天内回复
查看 CosmoStat/shapepipe 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 86/100
-
Validate finite output clamping limits in StepPredictor.prepare_clamping_params to prevent silent NaN/Inf propagation可能已有人在做 @muskan-creates352 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 84/100
mllam/neural-lam#848 ·
维护者通常 2 天内回复
-
infra
难度 1/5 1 小时以内 新手友好度 88/100
fab-geocommuns/RNB-coeur#1166 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
deepspeedai/DeepSpeed#8951 ·
维护者通常 2 天内回复