[BUG] Mis-typed free_blocks

Open Beginner friendly
#5,407 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
62/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
python

Research direction

Start with deepspeed/inference/v2/engine_v2.py at the reported uses around lines 200 and 214, then compare the free_blocks handling with get_kv_requirements in deepspeed/inference/v2/model_implementations/inference_transformer_base.py around lines 348 and 366-367. Run the provided engine.put reproduction and confirm it completes without the reported TypeError and returns logits.

Written by the indexing model from the issue text.

Description

bug inference

Describe the bug
The free_blocks variable is misused. It seems to be a list[int], but is sometimes used as an int, causing the program to crash.

It is correctly used in deepspeed/inference/v2/model_implementations/inference_transformer_base.py
https://github.com/microsoft/DeepSpeed/blob/a8b821535aa0b254efa681d51b4951734ca021cc/deepspeed/inference/v2/model_implementations/inference_transformer_base.py#L366-L367

But incorrectly used in deepspeed/inference/v2/engine_v2.py
https://github.com/microsoft/DeepSpeed/blob/a8b821535aa0b254efa681d51b4951734ca021cc/deepspeed/inference/v2/engine_v2.py#L200
https://github.com/microsoft/DeepSpeed/blob/a8b821535aa0b254efa681d51b4951734ca021cc/deepspeed/inference/v2/engine_v2.py#L214

To Reproduce

from mii.config import ModelConfig
from mii.modeling.models import load_model
from mii.modeling.tokenizers import load_tokenizer

config = ModelConfig(model_name_or_path="/path/to/llama-2-7b-hf", max_length=4096)
engine = load_model(config)
model = engine.model()
tokenizer = load_tokenizer(config)

tokens = tokenizer.encode("It is a period of civil wars in the galaxy. A brave alliance")

engine.put(batch_uids=[128], batch_tokens=[tokens])

When run, it will crash with:

Traceback (most recent call last):
  File "/tmp/tmp.py", line 86, in <module>
    main()
  File "/tmp/tmp.py", line 60, in main
    engine.put(batch_uids=[128], batch_tokens=[tokens])
  File "/path/conda/ds/lib/python3.11/site-packages/deepspeed/inference/v2/engine_v2.py", line 124, in put
    schedule_check = self.can_schedule(batch_uids, token_lens)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/conda/ds/lib/python3.11/site-packages/deepspeed/inference/v2/engine_v2.py", line 214, in can_schedule
    sched_len, sched_blocks = self._model.get_kv_requirements(seq_desc, length, free_blocks)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/conda/ds/lib/python3.11/site-packages/deepspeed/inference/v2/model_implementations/inference_transformer_base.py", line 348, in get_kv_requirements
    if block_lim <= max_new_blocks:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<=' not supported between instances of 'int' and 'list'

Expected behavior
engine.put should return the logits.

ds_report output
TODO

Screenshots
TODO

System info (please complete the following information):
TODO

Docker context
TODO

Additional context
TODO

Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 19h
Merged PRs (30d)
116

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deepspeedai/DeepSpeed

All issues in deepspeedai/DeepSpeed

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.