[BUG] Mis-typed free_blocks
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 62/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start with deepspeed/inference/v2/engine_v2.py at the reported uses around lines 200 and 214, then compare the free_blocks handling with get_kv_requirements in deepspeed/inference/v2/model_implementations/inference_transformer_base.py around lines 348 and 366-367. Run the provided engine.put reproduction and confirm it completes without the reported TypeError and returns logits.
Written by the indexing model from the issue text.
Description
Describe the bug
The free_blocks variable is misused. It seems to be a list[int], but is sometimes used as an int, causing the program to crash.
It is correctly used in deepspeed/inference/v2/model_implementations/inference_transformer_base.py
https://github.com/microsoft/DeepSpeed/blob/a8b821535aa0b254efa681d51b4951734ca021cc/deepspeed/inference/v2/model_implementations/inference_transformer_base.py#L366-L367
But incorrectly used in deepspeed/inference/v2/engine_v2.py
https://github.com/microsoft/DeepSpeed/blob/a8b821535aa0b254efa681d51b4951734ca021cc/deepspeed/inference/v2/engine_v2.py#L200
https://github.com/microsoft/DeepSpeed/blob/a8b821535aa0b254efa681d51b4951734ca021cc/deepspeed/inference/v2/engine_v2.py#L214
To Reproduce
from mii.config import ModelConfig
from mii.modeling.models import load_model
from mii.modeling.tokenizers import load_tokenizer
config = ModelConfig(model_name_or_path="/path/to/llama-2-7b-hf", max_length=4096)
engine = load_model(config)
model = engine.model()
tokenizer = load_tokenizer(config)
tokens = tokenizer.encode("It is a period of civil wars in the galaxy. A brave alliance")
engine.put(batch_uids=[128], batch_tokens=[tokens])
When run, it will crash with:
Traceback (most recent call last):
File "/tmp/tmp.py", line 86, in <module>
main()
File "/tmp/tmp.py", line 60, in main
engine.put(batch_uids=[128], batch_tokens=[tokens])
File "/path/conda/ds/lib/python3.11/site-packages/deepspeed/inference/v2/engine_v2.py", line 124, in put
schedule_check = self.can_schedule(batch_uids, token_lens)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/conda/ds/lib/python3.11/site-packages/deepspeed/inference/v2/engine_v2.py", line 214, in can_schedule
sched_len, sched_blocks = self._model.get_kv_requirements(seq_desc, length, free_blocks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/conda/ds/lib/python3.11/site-packages/deepspeed/inference/v2/model_implementations/inference_transformer_base.py", line 348, in get_kv_requirements
if block_lim <= max_new_blocks:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<=' not supported between instances of 'int' and 'list'
Expected behavior
engine.put should return the logits.
ds_report output
TODO
Screenshots
TODO
System info (please complete the following information):
TODO
Docker context
TODO
Additional context
TODO
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 19h
- Merged PRs (30d)
- 116
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from deepspeedai/DeepSpeed
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepspeedai/DeepSpeed#8577 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
deepspeedai/DeepSpeed#8514 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepspeedai/DeepSpeed#8500 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
deepspeedai/DeepSpeed#8371 ·
-
Difficulty 1/5 1-3 hours Newbie friendliness 68/100
deepspeedai/DeepSpeed#8101 · 1 comment ·
All issues in deepspeedai/DeepSpeed
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100