lm-sys/FastChat

Issues with VLLM Integration Speedup

Open

#2,362 opened on Sep 5, 2023

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Python (38,959 stars) (4,736 forks)batch import
good first issue

Description

Hello,

I've been trying to work with the [vLLM integration] and I'm facing some performance discrepancies. According to the documentation, I should achieve a significant speedup, but in my tests, I'm seeing different results:

Directly running with FastChat: 16 t/s Using the VLLM integration: 25 t/s (only 1.5x speedup) VLLM offline inference: 90 t/s (expected 6x speedup) I'm running Vicuna33b on a gin H100 gpu. Has anyone experienced this before? Are there any additional configurations or tweaks I might be missing to get the desired speedup?

Thanks in advance for any guidance or advice!

Contributor guide

Tech stack
python
Domain
performancebackend
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
PythonFastChat basicsvLLM integration
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
Investigate the vLLM integration code in FastChat, specifically the configuration and how it differs from offline vLLM inference. Compare the settings used in FastChat (e.g., in `fastchat/serve/vllm worker.py`) with the offline example. Check the documentation at `docs/vllm integration.md` for any missing flags. Look at the environment setup (e.g., CUDA, model loading) that might cause the performance gap. Also, search for similar issues in the FastChat repo for hints.