倉庫指標
- Star
- (26,687 star)
- PR 合併指標
- (平均合併 9天 15小時) (30 天內合併 3 個 PR)
描述
🐛 Bug
This might have a really obvious solution that I haven't found, but how is one supposed to handle two simultaneous SLURM jobs with automatic restart? I submit them both from the same directory. After a (long) time, job 0 begins and at some point later the job 1 begins. job 0 is eventually interrupted and saves a checkpoint hpc_ckpt_0, and subsequently job 1 is interrupted and saves hpc_ckpt_1. Now they're both requeued. But of course job 0 runs first and will load the latest checkpoint hpc_ckpt_1.
Is there some more sophisticated solution to this other than just running from different directories? Is there anywhere in the CheckpointConnector that would make sense to handle this case, or is it too niche?
Expected Behaviour
If two jobs are interrupted, they should have some way of associating the checkpoint to the job. Possibly by submitting the name of the checkpoint in the requeue of the job. Maybe this makes more sense as a feature request?
Version
I'm running PyLightning v1.5.10.
cc @borda @awaelchli