Epoch checkpoints can't be resumed once the next epoch has recropped (provider-transform-cache)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start with simpletuner/helpers/multiaspect/sampler.py at MultiAspectSampler.load_states, then trace DatasetRuntime.prepare_epoch, refresh_epoch_geometry, and the epoch checkpoint logic in trainer.py. Reproduce the resume failure using the supplied random recrop setup and epoch checkpoint interval. Done means both completed and interrupted rollovers resume safely, loaded latents match their records, and native Webshart checkpoints cover the same cases.
Written by the indexing model from the issue text.
Description
Summary
On refactor/provider-transform-cache (931460c57), a non-native dataset that recrops each epoch can't resume from an epoch checkpoint once the next epoch's rollover has started. The resume is refused at startup:
ValueError: Legacy consumed checkpoint requires unchanged prefetch, dataset, schedule and layout.
This hits anyone who uses random recrops with vae_cache_clear_each_epoch and relies on --checkpoint_epoch_interval. A stop anywhere in epoch N+1 before its first step checkpoint leaves only the epoch-N checkpoint, and that checkpoint can no longer be used.
Reproduction
- Dataset: local SDXL dataset, 21 images,
crop: true,crop_style: random,crop_aspect: random,crop_aspect_buckets: [1.0, 1.5, 0.67],vae_cache_clear_each_epoch: true,repeats: 1. - Run settings:
--checkpoint_epoch_interval 1,--checkpoint_step_interval 1000,--max_train_steps 30.
| Scenario | provider-transform-cache |
main (926e42ac4) |
refactor/fable-5.2-datasets |
|---|---|---|---|
| Run reaches epoch 2 (rollover completes), then resume from the epoch-1 checkpoint to step 40 | refused (error above) | crashes during training: Aspect ratios must be the same for all images in a batch |
resumes 40/40 |
| Process stopped inside epoch 2's rollover (new geometry saved, latents not yet refreshed), then resume | refused (error above) | not tested | not tested |
| Step checkpoint taken after the rollover, then resume | resumes 40/40 | not tested | not tested |
Each latent that training loaded was compared with the target_size in its record: 0 mismatches in every run that trained.
Cause
MultiAspectSampler.load_states (simpletuner/helpers/multiaspect/sampler.py:343-351) requires the checkpoint's aspect_ratio_bucket_indices to equal the metadata backend's current lists.
- An epoch checkpoint is written at the end of epoch N, recording epoch N+1 (
trainer.py:7701). Its sampler state holds epoch N's bucket lists. - The rollover into N+1 (
DatasetRuntime.prepare_epoch→refresh_epoch_geometry) rebuilds the geometry and saves the new lists. - On resume the saved cache therefore holds epoch N+1's buckets. In the run above, 13 of 21 samples had moved between buckets. Meanwhile the first
_epoch_rolloverafter the resume skipsprepare_epoch, becausefirst_epoch == epoch.
The refusal is safe, since it doesn't train on stale latents. But it leaves the epoch checkpoint unusable.
Direction
An epoch checkpoint for epoch N+1 with nothing consumed doesn't need epoch N's bucket lists. It should resume against whatever geometry epoch N+1 has: the saved cache if the rollover finished, otherwise a rollover redone on resume. A stop inside the rollover has to be detectable, so the resume either completes the rollover or rebuilds it, rather than mixing the new geometry with the old latents.
On refactor/fable-5.2-datasets, a checkpoint stores each bucket's consumed position and the epoch, not the bucket lists. These commits would need adapting to the provider's sampler:
- 447ab9b72:
DatasetCursor, a seeded per-rank traversal whose state is the number of samples consumed per bucket. - 86a5e5fa9: bucket lists stay whole on every rank, and the rank's share comes from the cursor.
- 33cb67231: a resumed epoch continues the uninterrupted run's batch schedule.
- dabf48f94: with prefetch, the checkpoint saves the position training has consumed.
Acceptance
- Resuming from an epoch checkpoint after a completed rollover trains epoch N+1 on its own geometry, with every loaded latent matching its record.
- A stop inside the rollover either resumes correctly or is refused with a message that names the cause and the fix. It must not train on mixed geometry.
- Native Webshart checkpoints (
native_epochstate) are tested for the same scenarios.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 295
- Avg merge
- 3h 28m
- Merged PRs (30d)
- 42
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from bghira/SimpleTuner
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
bghira/SimpleTuner#3345 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 57/100
bghira/SimpleTuner#3346 ·
Maintainers usually reply within 1 day
-
bug webui
Difficulty 3/5 1-2 days Newbie friendliness 72/100
bghira/SimpleTuner#3342 ·
Maintainers usually reply within 1 day
-
Parquet captions: skip null or empty rows instead of refusing to start, and never train on "nan"Openbug
Difficulty 4/5 3-5 days Newbie friendliness 55/100
bghira/SimpleTuner#3341 · 1 comment ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
bghira/SimpleTuner#3340 ·
Maintainers usually reply within 1 day
All issues in bghira/SimpleTuner
Similar issues
-
documentation priority: low size: S status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
cubrid-lab/sqlalchemy-cubrid#1397 ·
Maintainers usually reply within 1 day
-
documentation priority: low size: S status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
cubrid-lab/pycubrid#1307 ·
Maintainers usually reply within 1 day
-
[Bug]: Installer hides SSL certificate errors as bare URLError, and the suggested fix can't workOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
ShaftHQ/chaosengine-harness#143 ·
Maintainers usually reply within 1 day
-
LLMContextNode tool builders fail on nested params like `params:retrieval.max_matches`Possibly taken @Shizoqua claimed this today. OpenCommunity
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 2 days
-
security
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
btclib-org/portanode#985 ·
Maintainers usually reply within 1 day