Race condition with SLURM restart
#12.369 aberto em 17 de mar. de 2022
Métricas do repositório
- Stars
- (26.687 stars)
- Métricas de merge de PR
- (Mesclagem média 9d 15h) (3 fundiu PRs em 30d)
Description
🐛 Bug
This might have a really obvious solution that I haven't found, but how is one supposed to handle two simultaneous SLURM jobs with automatic restart? I submit them both from the same directory. After a (long) time, job 0 begins and at some point later the job 1 begins. job 0 is eventually interrupted and saves a checkpoint hpc_ckpt_0, and subsequently job 1 is interrupted and saves hpc_ckpt_1. Now they're both requeued. But of course job 0 runs first and will load the latest checkpoint hpc_ckpt_1.
Is there some more sophisticated solution to this other than just running from different directories? Is there anywhere in the CheckpointConnector that would make sense to handle this case, or is it too niche?
Expected Behaviour
If two jobs are interrupted, they should have some way of associating the checkpoint to the job. Possibly by submitting the name of the checkpoint in the requeue of the job. Maybe this makes more sense as a feature request?
Version
I'm running PyLightning v1.5.10.
cc @borda @awaelchli