Lightning-AI/pytorch-lightning

Provide a non-expert mode for the DeviceStatsMonitor

Open

#18,652 opened on Sep 27, 2023

View on GitHub
 (0 comments) (15 reactions) (0 assignees)Python (3,233 forks)batch import
callback: device statsfeaturehelp wanted

Repository metrics

Stars
 (26,687 stars)
PR merge metrics
 (Avg merge 9d 15h) (3 merged PRs in 30d)

Description

Description & Motivation

The DeviceStatsMonitor currently dumps the entire torch.cuda.memory_stats into the logger. Beginner users might be overwhelmed by the amount of data logged and are confused which metrics matter to maximize their utilization.

Pitch

Make the callback more user-friendly by logging only core metrics like GPU utilization and total memory usage. Provide an expert mode that when enabled logs what we see today.

Alternatives

No response

Additional context

This was proposed by user magical_arjay on our Discord.

cc @borda @awaelchli

Contributor guide

Research direction
Examine the current logging of torch.cuda.memory stats in DeviceStatsMonitor, design a configuration flag to select between expert and simplified logging, then implement the filtering of core metrics.
Tech stack
pythonpytorch
Domain
backendperformanceobservability
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Python
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75