Feature request: support for multi-node, multi-GPU distributed inference
Maintainers usually reply within 1 day
Nobody has claimed this yet.
- #4066 by @Saurav-Gupta-9741 — closed without merging
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Domain
- distributed-systems, machine-learning
Research direction
The issue names device_map="auto" and the goal of multi-node, multi-GPU inference, but it does not identify files, tests, or an implementation entry point. Start by tracing the existing device_map="auto" behavior and distributed launch path in Accelerate. Done would require a defined design and working inference across multiple nodes, including coverage for the proposed hardware configurations.
Written by the indexing model from the issue text.
Description
Motivation
Currently, device_map="auto" only support a single-node, multi-GPU setup (https://github.com/huggingface/transformers/issues/24747). If you have access to 8xA100 80GB/40GB, things are mostly fine, but not everyone has access to such a setup. Therefore, I believe there is benefit to support multi-node, multi-gpu inference. For example, if someone has access to multi-node 4xV100 setup but not 8xA100, then this feature would allow them to experiment large models like Llama-2-80B on their multi-node 4xV100 setup. As far as I know, there doesn't appear to be a straightforward method for running inference with models of the scale of Llama-2-80B on multi-node V100 setups.
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 1.6k
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 13
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/accelerate
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/accelerate#4511 ·
Maintainers usually reply within 1 day
-
BatchSamplerShard padding mutates retained sampler batches and changes original loader samplesPossibly taken @davrens101 claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
huggingface/accelerate#4505 ·
Maintainers usually reply within 1 day
-
accelerator.reduce(tensor, "mean") fails on integer tensors when more than one process is usedPossibly taken @MaiAditya claimed this 11 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/accelerate#4469 ·
Maintainers usually reply within 1 day
-
merge_fsdp_weights with remove_checkpoint_dir=True deletes the merged model when output_path is inside checkpoint_dirPossibly taken @assass1nlike claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
huggingface/accelerate#4514 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
huggingface/accelerate#4510 ·
Maintainers usually reply within 1 day
All issues in huggingface/accelerate
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
bigbio/sdrf-pipelines#372 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 84/100
521xueweihan/HelloGitHub#4297 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
aws-samples/sample-per-model-bedrock#822 ·
Maintainers usually reply within 1 day
-
area:study-data p2-high ready-for-agent type:chore
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
fderuiter/deruiter.dev#3706 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
awslabs/visual-asset-management-system#1234 ·
Maintainers usually reply within 1 day