Lightning-AI/pytorch-lightning

If ModelCheckpoint and Validation occur on the same step, validation should run first, reopen

Open

#17,417 opened on Apr 19, 2023

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (3,233 forks)batch import
bugcallback: model checkpointhelp wantedrepro needed

Repository metrics

Stars
 (26,687 stars)
PR merge metrics
 (Avg merge 9d 15h) (3 merged PRs in 30d)

Description

Bug description

The issue described here, is still unsolved https://github.com/Lightning-AI/lightning/issues/7694

The consequences of this issue is that n-th model is saved based on metrics from the n-1th step

What version are you seeing the problem on?

No response

How to reproduce the bug

No response

Error messages and logs

# Error messages and logs here please

Environment

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):

More info

No response

cc @carmocca @awaelchli

Contributor guide

Research direction
Investigate the order of callback execution in the training loop and ensure that validation runs before checkpointing when they are scheduled on the same step.
Tech stack
pythonpytorch
Domain
ai
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Python
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70