Lightning-AI/pytorch-lightning

Resuming should allow to differentiate what to resume (steps/opti/weights)

Open

#5,339 opened on Jan 3, 2021

View on GitHub
 (26 comments) (18 reactions) (0 assignees)Python (3,233 forks)batch import
featurehelp wantedpriority: 1

Repository metrics

Stars
 (26,687 stars)
PR merge metrics
 (Avg merge 9d 15h) (3 merged PRs in 30d)

Description

Currently it is possible to either resume only the full training state (epoch/global steps / optimizer / scheduler options / and weights), or only the weights.

I would like to be able to switch the optimizer at some point, i.e. skip restoring optimizer/scheduler, but still load the epoch/global steps. I only see a way to do this with hacks at the moment. Any other way? Could this be a feature, to specify in the trainer.init function specifically what to restore/what not to restore?

Contributor guide

Research direction
Investigate the current checkpoint resume logic in Lightning and modify the `Trainer` or `CheckpointConnector` to accept a parameter for selective restoration of training state components.
Tech stack
python
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonPyTorch Lightning
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40