enhancementgood first issue
仓库指标
- Star
- (38,959 star)
- PR 合并指标
- (30 天内没有已合并 PR)
描述
According to some recent analysis on twitter, CTranslate2 can serve LLMs a little faster than vLLM and (maybe?) with a small quality increase. At least for Llama 2.
This could either be a a model worker that's added directly to fastchat OR a doc with extensive documentation on how to write a custom model worker (with mostly working \ implementation code) that anyone can use in their own project. The second option might be best so that people can expand FastChat with custom model workers without having to change the base project too much.
If this is appealing I can get to it at some point.