Loading weights straight to GPU only works for inference
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
Research direction
Start with the official example scripts in the examples/ folder, including the supported no_trainer scripts such as run_no_trainer_glue.py, and review the documented weight-loading path. Verify the behavior for training in both single-GPU and multi-GPU cases; done means weights can load directly to the GPU for both configurations.
Written by the indexing model from the issue text.
Description
System Info
accelerate v0.22.0
Information
- The official example scripts
- My own modified scripts
Tasks
- One of the scripts in the examples/ folder of Accelerate or an officially supported
no_trainerscript in theexamplesfolder of thetransformersrepo (such asrun_no_trainer_glue.py) - My own task or dataset (give details below)
Reproduction
Going through the doc, accelerate currently does not seem to support loading weights straight to the GPU if the model is to be used for training. If this is correct, are there any plans to support this in the future?
Expected behavior
It would be nice to be able to load weights straight to GPU when training, and to be able to do this either in the single or multi-gpu case.
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 1.6k
- Avg merge
- 20h 23m
- Merged PRs (30d)
- 12
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/accelerate
-
FSDP2: prepare() keeps the whole unsharded model alive during sharding when cpu_ram_efficient_loading=FalsePossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/accelerate#4511 ·
Maintainers usually reply within 1 day
-
BatchSamplerShard padding mutates retained sampler batches and changes original loader samplesPossibly taken @davrens101 claimed this 4 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
huggingface/accelerate#4505 ·
Maintainers usually reply within 1 day
-
accelerator.reduce(tensor, "mean") fails on integer tensors when more than one process is usedPossibly taken @MaiAditya claimed this 13 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/accelerate#4469 ·
Maintainers usually reply within 1 day
-
merge_fsdp_weights with remove_checkpoint_dir=True deletes the merged model when output_path is inside checkpoint_dirPossibly taken @assass1nlike claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
huggingface/accelerate#4514 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
huggingface/accelerate#4510 ·
Maintainers usually reply within 1 day
All issues in huggingface/accelerate
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
SACGF/variantgrid#2791 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
bot-triaged oncall: cpu inductor
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
kubescape/cel-admission-library#141 · 1 comment ·
Maintainers usually reply within 1 day