lm-sys/FastChat

tiiuae/falcon-7b does not work on Apple M1 GPU (MPS)

Open

#1,746 opened on 2023年6月20日

GitHub で見る
 (2 comments) (1 reaction) (0 assignees)Python (38,959 stars) (4,736 forks)batch import
good first issue

説明

Hey,

I am trying to run

python3 -m fastchat.serve.cli --model-path tiiuae/falcon-7b --device mps
tiiuae/falcon-7b

Prompt: User: Who is King Charles?

This leads to the following error:

Assistant: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/transformers/generation/utils.py:723: UserWarning: MPS: no support for int64 repeats mask, casting it to int32 (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/mps/operations/Repeat.mm:236.)
  input_ids = input_ids.repeat_interleave(expand_size, dim=0)
loc("varianceEps"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/c2cb9645-dafc-11ed-aa26-6ec1e3b3f7b3/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":228:0)): error: input types 'tensor<1x12x1xf16>' and 'tensor<1xf32>' are not broadcast compatible
LLVM ERROR: Failed to infer result type(s).
[1]    64954 abort      python3 -m fastchat.serve.cli --model-path tiiuae/falcon-7b --device mps
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Is there anything I am missing? Thanks!

コントリビューターガイド

技術スタック
pythonpytorch
領域
machine learningbackend
Issue 種別
bug
難度新しい貢献者にとっての実装難度です。1 は非常に小さな変更、5 は専門的な作業です。
4
推定時間経験ある貢献者が調査、実装、テスト、pull request 準備にかけるおおよその時間範囲です。
1-2 days
活動状況Issue が今どれだけ取り組みやすいかを示します。新しい、活発、古い、ブロック中、またはメンテナー入力待ちなどです。
fresh
明確さ期待される変更、受け入れ条件、次の手順がどれだけ明確かを示します。
clear
前提条件
PythonPyTorchApple M1 Mactransformers library
初心者向け度初回貢献者にどれだけ取り組みやすいかを 1-100 で推定したスコアです。
40
調査方針
The error occurs in the HuggingFace transformers generation code when using MPS device. The issue is likely due to a dtype mismatch (float16 vs float32) during the repeat interleave operation. To fix, investigate the relevant code in transformers/src/transformers/generation/utils.py around line 723 and consider casting tensors to a common dtype before the operation. Testing on an Apple M1 Mac is required.