Confusion about Deepspeed Inference
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Documentation
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- ai, machine-learning
Research direction
The issue names deepspeed.initialize, deepspeed.init_inference, MII, Zero-3, CPU offload, and 70B LLaMA inference. Start by comparing the documented inference paths and their memory and model-parallelism behavior; done would be a documented answer covering the three questions and current best practices.
Written by the indexing model from the issue text.
Description
Hi, I read the deepspeed docs and have the following confusion:
(1) What's the difference between these methods when in inferencing LLMs?
a. deepspeed.initialize and then write code to generate text
b. deepspeed.init_inference then write code to generate
c. use mii to inference
(2) Which of them are friendly for memory? For example, I want to inference 70b models, which of them support model parallelism that separates model parameters across gpus?
(3) For inference, what's the best practice now for inferencing 70b llama?
a. zero3 + cpu offload (1*a100)
b. zero3 (2*a100)
...
Thank you!
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from deepspeedai/DeepSpeedExamples
-
Difficulty 2/5 1-3 hours Newbie friendliness 52/100
deepspeedai/DeepSpeedExamples#996 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
deepspeedai/DeepSpeedExamples#995 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 52/100
deepspeedai/DeepSpeedExamples#989 ·
-
moe example 404 Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
deepspeedai/DeepSpeedExamples#984 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
deepspeedai/DeepSpeedExamples#979 · 6 comments ·
All issues in deepspeedai/DeepSpeedExamples
Similar issues
-
area: harness bug status: needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Human-Agent-Society/reef#625 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 1/5 Under an hour Newbie friendliness 80/100
learningequality/kolibri#15351 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Name consistency Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
eellak/triplestore#65 · 1 comment ·