lm-sys/FastChat

Issues with VLLM Integration Speedup

Open

#2,362 创建于 2023年9月5日

在 GitHub 查看
 (4 评论) (0 反应) (0 负责人)Python (38,959 star) (4,736 fork)batch import
good first issue

描述

Hello,

I've been trying to work with the [vLLM integration] and I'm facing some performance discrepancies. According to the documentation, I should achieve a significant speedup, but in my tests, I'm seeing different results:

Directly running with FastChat: 16 t/s Using the VLLM integration: 25 t/s (only 1.5x speedup) VLLM offline inference: 90 t/s (expected 6x speedup) I'm running Vicuna33b on a gin H100 gpu. Has anyone experienced this before? Are there any additional configurations or tweaks I might be missing to get the desired speedup?

Thanks in advance for any guidance or advice!

贡献者指南

技术栈
python
领域
performancebackend
议题类型
bug
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
4
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
half day
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
stale
清晰度议题是否清楚说明期望改动、验收标准和下一步。
mostly clear
前置要求
PythonFastChat basicsvLLM integration
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
35
研究方向
Investigate the vLLM integration code in FastChat, specifically the configuration and how it differs from offline vLLM inference. Compare the settings used in FastChat (e.g., in `fastchat/serve/vllm worker.py`) with the offline example. Check the documentation at `docs/vllm integration.md` for any missing flags. Look at the environment setup (e.g., CUDA, model loading) that might cause the performance gap. Also, search for similar issues in the FastChat repo for hints.
Issues with VLLM Integration Speedup · lm-sys/FastChat#2362 | Good First Issue