HPC Resubmit resume on most recent epoch checkpoint

🚀 Feature

Allow end-of-epoch checkpoints for resuming killed and resubmitted training jobs in a SLURM environment.

Motivation

Mid-epoch checkpointing does not appear to work with my model, even with fault-tolerant training I still get some weird results. Since I am training on a smaller dataset with a larger number of epochs, it would be really useful for me to be able to resume from the most recent checkpoint I saved using the normal end-of-epoch checkpoints.

Pitch

Instead of forcing users into the checkpoint process defined by the SLURMEnvironment plugin, allowing user's to customize the pause/resume operation would be a useful feature. Maybe add this as an option to the SLURMEnvironment plugin? Since my SLURM job ID is the same after resubmission, the default_root_dir is being set to the same as the previous job so the newest checkpoint should be easy to find.

Alternatives

Just an option for resuming end-of-epoch checkpoints would solve my problem. Allowing hooks for full customization of this function would be the most customizable version but put the most work on the user.

cc @borda @awaelchli @ananthsub @ninginthecloud @rohitgr7 @otaj