RVC-Project/Retrieval-based-Voice-Conversion-WebUI

Error in epoch rounds, recounting from 1

Open

#1,954 opened on Apr 7, 2024

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (18,427 stars) (2,849 forks)batch import
help wanted

Description

During the training process of my model, because I did not reserve enough space on the hard drive, the model stopped with an error "OSError:No space left on device". I cleaned it up and left enough space to start training again, but it should have trained for more than 400 rounds, but the epoch is counting from 1. Is this because the checkpoints are missing? Is there any negative impact on my training? I really need to solve this problem urgently, thanks for your help!

Contributor guide

Tech stack
python
Domain
machine learning
Issue type
research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
unclear
Prerequisites
basic knowledge of training resumptionfamiliarity with checkpointing
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Investigate the training script's checkpoint loading logic, particularly the epoch counter. The user experienced a training interruption due to disk space, and after cleanup, training restarted from epoch 1. Check if the checkpoint file was corrupted or missing. Look at the training loop code to understand how epoch state is saved and restored. Provide guidance on how to resume training from the last saved epoch.