lm-sys/FastChat

support for 4bit quantization from transfomer library.

Open

#1,798 创建于 2023年6月27日

在 GitHub 查看
 (7 评论) (2 反应) (0 负责人)Python (38,959 star) (4,736 fork)batch import
enhancementgood first issue

描述

Loading a vicuna13B using 4bit quantization from the transformers library is possible load_in_4bit. How difficult could be for Fastach to support it?

贡献者指南

技术栈
pythonpytorch
领域
machine learningai
议题类型
feature
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
4
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
1-2 days
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
needs maintainer response
清晰度议题是否清楚说明期望改动、验收标准和下一步。
mostly clear
前置要求
PythonPyTorchHugging Face TransformersQuantization basics
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
50
研究方向
Investigate how FastChat currently loads models (e.g., in model worker.py) and replicate the load in 4bit parameter from the transformers library. Review the transformers quantization documentation linked in the issue to understand the API. Check existing comments for any insights or blockers. Implement the parameter in the model loading pipeline, ensuring compatibility with the existing model serving infrastructure.
support for 4bit quantization from transfomer library. · lm-sys/FastChat#1798 | Good First Issue