RVC-Project/Retrieval-based-Voice-Conversion-WebUI

Training is freezing the computer when training for more than 20 epochs.

Open

#1,711 opened on Jan 14, 2024

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (18,427 stars) (2,849 forks)batch import
AMD/Intelhelp wanted

Description

Are there any options I can add to the invoking command to limit the VRAM usage. Like the way Automatic1111 for Stable Diffusion has --lowvram option.

Because when I'm trying to train a model for more than 20 epochs, it is freezing the entire computer. I have RX 6500 XT with 4GB, Main RAM is 16GB.

Contributor guide

Tech stack
pythonpytorch
Domain
machine learningperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
PythonPyTorchmemory managementCUDA programming
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
The issue requests a lowvram option to limit VRAM usage during training, similar to Automatic1111's option. To implement, investigate the training script (likely in `train.py` or `infer.py`) to identify where model and data batches are loaded. Consider adding a command line argument using argparse that enables memory saving techniques like gradient checkpointing, mixed precision, or reducing batch size dynamically. Check the repository's existing argument parsing for guidance.
Training is freezing the computer when training for more than 20 epochs. · RVC-Project/Retrieval-based-Voice-Conversion-WebUI#1711 | Good First Issue