Race condition with SLURM restart

🐛 Bug

This might have a really obvious solution that I haven't found, but how is one supposed to handle two simultaneous SLURM jobs with automatic restart? I submit them both from the same directory. After a (long) time, job 0 begins and at some point later the job 1 begins. job 0 is eventually interrupted and saves a checkpoint hpc_ckpt_0, and subsequently job 1 is interrupted and saves hpc_ckpt_1. Now they're both requeued. But of course job 0 runs first and will load the latest checkpoint hpc_ckpt_1.

Is there some more sophisticated solution to this other than just running from different directories? Is there anywhere in the CheckpointConnector that would make sense to handle this case, or is it too niche?

Expected Behaviour

If two jobs are interrupted, they should have some way of associating the checkpoint to the job. Possibly by submitting the name of the checkpoint in the requeue of the job. Maybe this makes more sense as a feature request?

Version

I'm running PyLightning v1.5.10.

cc @borda @awaelchli