Lightning-AI/pytorch-lightning

Redesign dataloader re-instantiation context managers for Lite

Open

#15,179 opened on Oct 18, 2022

View on GitHub
 (10 comments) (0 reactions) (1 assignee)Python (3,233 forks)batch import
discussionfabrichelp wantedpriority: 2refactor

Repository metrics

Stars
 (26,687 stars)
PR merge metrics
 (Avg merge 9d 15h) (3 merged PRs in 30d)

Description

Proposed refactor

Redesign the context managers that enable re-instantiations of DataLoader after #14992 deprecates the run method design.

Motivation

We currently have two context managers over the run method:

https://github.com/Lightning-AI/lightning/blob/776432fd7e34cc02e9dbfb16ad8ec8491c176dbf/src/lightning_lite/lite.py#L394-L397

  • _replace_dunder_methods(DataLoader, "dataset")
  • _replace_dunder_methods(BatchSampler)

These get applied for the user if they implement the run method. However, this won't be the case anymore after #14992. We need to find an alternative approach.

Pitch

We probably need to call these methods at the start of the program, for example in LightningLite.__init__. But it is unclear how to undo the patching which is usually done in __exit__. Happy to hear suggestions.

cc @borda @justusschock @awaelchli @tchaton @carmocca @otaj

Contributor guide

Research direction
Review the two context managers ` replace dunder methods` in Lite, understand how they depend on the run method, and explore alternatives to apply them at initialization without needing an exit. Consider patterns like monkey patching at ` init ` or using a setup/teardown mechanism.
Tech stack
python
Domain
backenddevtools
Issue type
Refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
Pythoncontext managers
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40