[FSDP] support activation offloading with FSDP
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Domain
- distributed-systems, machine-learning
Research direction
Start with the linked torch/distributed/algorithms/_checkpoint/checkpoint_wrapper.py section and the apply_activation_checkpointing entry point. Determine how whole-model activation offloading should work with FSDP and activation checkpointing; done means the root module can be wrapped so activations are offloaded between layers and GPU memory usage is reduced.
Written by the indexing model from the issue text.
Description
Support whole model activation offloading with FSDP - working in conjunction with activation checkpointing - via
As apply_activation_checkpointing does not wrap the overall root module, wrapping the overall root module with this could offload activation between layer, thus release more GPU memory. The diff should be small and i am happy to work on this.
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 1.6k
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 13
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/accelerate
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/accelerate#4511 ·
Maintainers usually reply within 1 day
-
BatchSamplerShard padding mutates retained sampler batches and changes original loader samplesPossibly taken @davrens101 claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
huggingface/accelerate#4505 ·
Maintainers usually reply within 1 day
-
accelerator.reduce(tensor, "mean") fails on integer tensors when more than one process is usedPossibly taken @MaiAditya claimed this 11 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/accelerate#4469 ·
Maintainers usually reply within 1 day
-
merge_fsdp_weights with remove_checkpoint_dir=True deletes the merged model when output_path is inside checkpoint_dirPossibly taken @assass1nlike claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
huggingface/accelerate#4514 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
huggingface/accelerate#4510 ·
Maintainers usually reply within 1 day
All issues in huggingface/accelerate
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
bigbio/sdrf-pipelines#372 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
521xueweihan/HelloGitHub#4297 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
aws-samples/sample-per-model-bedrock#822 ·
Maintainers usually reply within 1 day
-
area:study-data p2-high ready-for-agent type:chore
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
fderuiter/deruiter.dev#3706 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
awslabs/visual-asset-management-system#1234 ·
Maintainers usually reply within 1 day