lm-sys/FastChat

Support fastchat-t5-3b-v1.0 on M2 GPU model

Open

#1,744 opened on Jun 20, 2023

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (38,959 stars) (4,736 forks)batch import
good first issue

Description

CPU model working fine python -m fastchat.serve.cli --model-path /path/to/fastchat-t5-3b-v1.0 --device cpu

GPU models both got errors python -m fastchat.serve.cli --model-path /path/to/fastchat-t5-3b-v1.0 --device mps --load-8bit python -m fastchat.serve.cli --model-path /path/to/fastchat-t5-3b-v1.0 --device mps

So, does fastchat-t5-3b-v1.0 can running on Mac --device mps?

Contributor guide

Tech stack
python
Domain
machine learning
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
needs maintainer response
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Basic knowledge of FastChatMPS device setup
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
The issue reports that fastchat t5-3b v1.0 fails on MPS GPU while CPU works. The error is likely in the model loading or device mapping code. Investigate the fastchat.serve.cli module and how it handles the device and load 8bit flags. The model may need specific quantization or MPS specific adjustments. Look for similar issues or patches in the FastChat repository. The user did not provide full error logs, so reproducing the error with the given commands is necessary.