Epoch checkpoints can't be resumed once the next epoch has recropped (provider-transform-cache)

Open
#3,343 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python

Research direction

Start with simpletuner/helpers/multiaspect/sampler.py at MultiAspectSampler.load_states, then trace DatasetRuntime.prepare_epoch, refresh_epoch_geometry, and the epoch checkpoint logic in trainer.py. Reproduce the resume failure using the supplied random recrop setup and epoch checkpoint interval. Done means both completed and interrupted rollovers resume safely, loaded latents match their records, and native Webshart checkpoints cover the same cases.

Written by the indexing model from the issue text.

Description

bug

Summary

On refactor/provider-transform-cache (931460c57), a non-native dataset that recrops each epoch can't resume from an epoch checkpoint once the next epoch's rollover has started. The resume is refused at startup:

ValueError: Legacy consumed checkpoint requires unchanged prefetch, dataset, schedule and layout.

This hits anyone who uses random recrops with vae_cache_clear_each_epoch and relies on --checkpoint_epoch_interval. A stop anywhere in epoch N+1 before its first step checkpoint leaves only the epoch-N checkpoint, and that checkpoint can no longer be used.

Reproduction

  • Dataset: local SDXL dataset, 21 images, crop: true, crop_style: random, crop_aspect: random, crop_aspect_buckets: [1.0, 1.5, 0.67], vae_cache_clear_each_epoch: true, repeats: 1.
  • Run settings: --checkpoint_epoch_interval 1, --checkpoint_step_interval 1000, --max_train_steps 30.
Scenario provider-transform-cache main (926e42ac4) refactor/fable-5.2-datasets
Run reaches epoch 2 (rollover completes), then resume from the epoch-1 checkpoint to step 40 refused (error above) crashes during training: Aspect ratios must be the same for all images in a batch resumes 40/40
Process stopped inside epoch 2's rollover (new geometry saved, latents not yet refreshed), then resume refused (error above) not tested not tested
Step checkpoint taken after the rollover, then resume resumes 40/40 not tested not tested

Each latent that training loaded was compared with the target_size in its record: 0 mismatches in every run that trained.

Cause

MultiAspectSampler.load_states (simpletuner/helpers/multiaspect/sampler.py:343-351) requires the checkpoint's aspect_ratio_bucket_indices to equal the metadata backend's current lists.

  • An epoch checkpoint is written at the end of epoch N, recording epoch N+1 (trainer.py:7701). Its sampler state holds epoch N's bucket lists.
  • The rollover into N+1 (DatasetRuntime.prepare_epoch → refresh_epoch_geometry) rebuilds the geometry and saves the new lists.
  • On resume the saved cache therefore holds epoch N+1's buckets. In the run above, 13 of 21 samples had moved between buckets. Meanwhile the first _epoch_rollover after the resume skips prepare_epoch, because first_epoch == epoch.

The refusal is safe, since it doesn't train on stale latents. But it leaves the epoch checkpoint unusable.

Direction

An epoch checkpoint for epoch N+1 with nothing consumed doesn't need epoch N's bucket lists. It should resume against whatever geometry epoch N+1 has: the saved cache if the rollover finished, otherwise a rollover redone on resume. A stop inside the rollover has to be detectable, so the resume either completes the rollover or rebuilds it, rather than mixing the new geometry with the old latents.

On refactor/fable-5.2-datasets, a checkpoint stores each bucket's consumed position and the epoch, not the bucket lists. These commits would need adapting to the provider's sampler:

  • 447ab9b72: DatasetCursor, a seeded per-rank traversal whose state is the number of samples consumed per bucket.
  • 86a5e5fa9: bucket lists stay whole on every rank, and the rank's share comes from the cursor.
  • 33cb67231: a resumed epoch continues the uninterrupted run's batch schedule.
  • dabf48f94: with prefetch, the checkpoint saves the position training has consumed.

Acceptance

  • Resuming from an epoch checkpoint after a completed rollover trains epoch N+1 on its own geometry, with every loaded latent matching its record.
  • A stop inside the rollover either resumes correctly or is refused with a message that names the cause and the fix. It must not train on mixed geometry.
  • Native Webshart checkpoints (native_epoch state) are tested for the same scenarios.
Dominant language
Python
Stars
2.9k
Forks
295
Avg merge
3h 28m
Merged PRs (30d)
42

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from bghira/SimpleTuner

All issues in bghira/SimpleTuner

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.