lm-sys/FastChat

[Feature Request] Add a ctranslate2 model worker

オープン

#2,133 opened on 2023/08/01

 (2 件のコメント) (1 件のリアクション) (1 人の担当者)Python (4,736 件のフォーク)batch import
enhancementgood first issue

Repository metrics

Stars
 (38,959 個のスター)
PR merge metrics
 (PR metrics pending)

説明

According to some recent analysis on twitter, CTranslate2 can serve LLMs a little faster than vLLM and (maybe?) with a small quality increase. At least for Llama 2.

This could either be a a model worker that's added directly to fastchat OR a doc with extensive documentation on how to write a custom model worker (with mostly working \ implementation code) that anyone can use in their own project. The second option might be best so that people can expand FastChat with custom model workers without having to change the base project too much.

If this is appealing I can get to it at some point.

コントリビューターガイド