ggml-org/llama.cpp
在 GitHub 查看Study how LM Evaluation Harness works and try to implement it
Open
#231 建立於 2023年3月17日
enhancementgeneration qualityhelp wantedhigh priorityresearch 🔬
倉庫指標
- Star
- (110,169 star)
- PR 合併指標
- (平均合併 6天 8小時) (30 天內合併 389 個 PR)
描述
Update 10 Apr 2024: https://github.com/ggerganov/llama.cpp/issues/231#issuecomment-2047759312
It would be great to start doing this kind of quantitative analysis of ggml-based inference:
https://bellard.org/ts_server/
It looks like Fabrice evaluates the models using something called LM Evaluation Harness:
https://github.com/EleutherAI/lm-evaluation-harness
I have no idea what this is yet, but would be nice to study it and try to integrate it here and in other ggml-based projects.
This will be very important step needed to estimate the quality of the generated output and see if we are on the right track.