lm-sys/FastChat

Issues with VLLM Integration Speedup

Open

#2,362 opened on 2023年9月5日

GitHub で見る
 (4 comments) (0 reactions) (0 assignees)Python (38,959 stars) (4,736 forks)batch import
good first issue

説明

Hello,

I've been trying to work with the [vLLM integration] and I'm facing some performance discrepancies. According to the documentation, I should achieve a significant speedup, but in my tests, I'm seeing different results:

Directly running with FastChat: 16 t/s Using the VLLM integration: 25 t/s (only 1.5x speedup) VLLM offline inference: 90 t/s (expected 6x speedup) I'm running Vicuna33b on a gin H100 gpu. Has anyone experienced this before? Are there any additional configurations or tweaks I might be missing to get the desired speedup?

Thanks in advance for any guidance or advice!

コントリビューターガイド

技術スタック
python
領域
performancebackend
Issue 種別
bug
難度新しい貢献者にとっての実装難度です。1 は非常に小さな変更、5 は専門的な作業です。
4
推定時間経験ある貢献者が調査、実装、テスト、pull request 準備にかけるおおよその時間範囲です。
half day
活動状況Issue が今どれだけ取り組みやすいかを示します。新しい、活発、古い、ブロック中、またはメンテナー入力待ちなどです。
stale
明確さ期待される変更、受け入れ条件、次の手順がどれだけ明確かを示します。
mostly clear
前提条件
PythonFastChat basicsvLLM integration
初心者向け度初回貢献者にどれだけ取り組みやすいかを 1-100 で推定したスコアです。
35
調査方針
Investigate the vLLM integration code in FastChat, specifically the configuration and how it differs from offline vLLM inference. Compare the settings used in FastChat (e.g., in `fastchat/serve/vllm worker.py`) with the offline example. Check the documentation at `docs/vllm integration.md` for any missing flags. Look at the environment setup (e.g., CUDA, model loading) that might cause the performance gap. Also, search for similar issues in the FastChat repo for hints.