Lightning-AI/pytorch-lightning

Change in logging frequency for automated logging

Open

#16.821 aperta il 20 feb 2023

Vedi su GitHub
 (3 commenti) (0 reazioni) (0 assegnatari)Python (3233 fork)batch import
bughelp wanted

Metriche repository

Star
 (26.687 star)
Metriche merge PR
 (Merge medio 9g 15h) (3 PR mergiate in 30 g)

Descrizione

Bug description

Utilising Automated Logging with self.log and self.log_dict as described in the documentation results in a shift of the logging frequency after various amounts of steps.

change_log_freq

This is also observed in all train metrics but only in the _step. It could be mutually exclusive to the WandbLogger but has already been observed to some extent in the TensorboardLogger as reported in #13525 and more generally in #10436.

How to reproduce the bug

Call self.log(, on_step=True, on_epoch=True) in training_step and let it run for more than 15k steps (in my case). The logging rate initial is equal to log_every_n_steps=50 for some iterations but jumps wildly around for others.

batch_size=10 (specified in self.log(batch_size=10)) test_subjects=20 samples_per_subject=10

This equals to 200 samples per epoch and 20 steps per epoch. Even if log_every_n_steps=50, this should log then precisely every 100 steps (as the first 50 is not met according to my understanding) and not jump around from 2 to 200.

Environment

  • Lightning Component: Trainer/ LightningModule
  • Python 3.9
  • Pytorch-lightning 1.9.1 (installed with pip)
  • PyTorch 1.13.1
  • CUDA/ cuDNN: cuda11.6-cudnn8
  • OS: Linux (Kernel 3.10.0-1160)
  • Running environment: server

Guida contributor