Feature request: ability to truncate datasets on split rather than pad
Maintainers usually reply within 1 day
Nobody has claimed this yet.
- #3615 by @ved1beta — closed without merging
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- data, distributed-systems
Research direction
Start by locating Accelerate’s built-in dataset splitting across processes and inspect how padding interacts with repeated samples when a seen list tracks them. Define the truncation behavior around batch_size * gradient_steps * num_processes, then verify that all processes finish without deadlocking and that sample tracking remains correct.
Written by the indexing model from the issue text.
Description
As a Developer working on latent diffusion model training via SimpleTuner, it has become evident that the built-in mechanism for splitting datasets across processes is not smart enough to apply in cases where a robust sample-tracking mechanism is in use.
SimpleTuner uses a 'seen' list to keep track of samples per epoch so that we do not inadvertently oversample. This has the side effect of padding not actually working, since the repeated samples in the list are simply discarded.
What happens next, is that one of the GPUs runs out of data just before the other would have, and then causes a deadlock while the main process waits for the backward pass, which will never come.
My solution was to truncate the sets I'm splitting, taking into account the batch_size * gradient_steps * num_processes and then split it. But, it occurred to me, having this be built-in would be nice.
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 1.6k
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 13
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/accelerate
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/accelerate#4511 ·
Maintainers usually reply within 1 day
-
BatchSamplerShard padding mutates retained sampler batches and changes original loader samplesPossibly taken @davrens101 claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
huggingface/accelerate#4505 ·
Maintainers usually reply within 1 day
-
accelerator.reduce(tensor, "mean") fails on integer tensors when more than one process is usedPossibly taken @MaiAditya claimed this 11 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/accelerate#4469 ·
Maintainers usually reply within 1 day
-
merge_fsdp_weights with remove_checkpoint_dir=True deletes the merged model when output_path is inside checkpoint_dirPossibly taken @assass1nlike claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
huggingface/accelerate#4514 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
huggingface/accelerate#4510 ·
Maintainers usually reply within 1 day
All issues in huggingface/accelerate
Similar issues
-
`destroy` sans terminal répond « Aborted. » au lieu de réclamer `--yes` comme `clean` et `reset`Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
stephrobert/dsoxlab#354 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
jasonacox/pypowerwall#472 ·
Maintainers usually reply within 1 day
-
[Bug]: DoubleMLRDDData accepts a score_col that is no data column (as in its docstring example); the running variable then becomes a covariate and RDFlex failsPossibly taken @vrjdev claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
DoubleML/doubleml-for-py#455 ·
Maintainers usually reply within 1 day
-
[Bug]: Python mod.Box[T] annotation gets a false generic_arg edge that replaces the parameter_type one ('is' on tree-sitter nodes)Possibly taken @rohit-jsfreaky claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Graphify-Labs/graphify#5110 · 1 comment ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
assafelovic/gpt-researcher#2277 ·
Maintainers usually reply within 10 days