[BUG] RWKV Models are not configured to use Cuda GPU lists. · lm-sys/FastChat#1248

(2 comments) (0 reactions) (0 assignees)Python (38,959 stars) (4,736 forks)batch import

buggood first issue

説明

Hello,

The file (fastchat/model/rwkv_model.py) has a RWKV stratergy (strategy="cuda fp16") being set that is not imlemented to use the native --gpus or --num-gpus fschat command arguments.

Example, Launching a model worker with (--num-gpus 4 --gpus 0,1,2,3) does not spawn the RWKV instance with these GPU specifications and just uses GPU 1

strategy="cuda fp16"

VS.

strategy="cuda:0 fp16 -> cuda:1 fp16 -> cuda:2 fp16 -> cuda:3 fp16 -> cuda:4 fp16"

コントリビューターガイド

技術スタック: python
領域: backendmachine learning
Issue 種別: bug
難度: 2
推定時間: 1-3 hours
活動状況: fresh
明確さ: mostly clear
前提条件: Basic PythonUnderstanding of CUDA GPU management
初心者向け度: 40
調査方針: The issue is in fastchat/model/rwkv model.py where the strategy is hardcoded as 'cuda fp16'. The model worker command line arguments num gpus and gpus are not being used to set the strategy accordingly. The fix should parse these arguments and construct a strategy string that specifies which GPUs to use, e.g., 'cuda:0 fp16 > cuda:1 fp16'. Refer to the model worker's argument parsing in fastchat/serve/model worker.py and the existing GPU allocation logic.