ggml-org/llama.cpp

Study how LM Evaluation Harness works and try to implement it

Open

#231 opened on Mar 17, 2023

View on GitHub
 (10 comments) (9 reactions) (0 assignees)C++ (18,202 forks)batch import
enhancementgeneration qualityhelp wantedhigh priorityresearch 🔬

Repository metrics

Stars
 (110,169 stars)
PR merge metrics
 (Avg merge 6d 8h) (389 merged PRs in 30d)

Description

Update 10 Apr 2024: https://github.com/ggerganov/llama.cpp/issues/231#issuecomment-2047759312

It would be great to start doing this kind of quantitative analysis of ggml-based inference:

https://bellard.org/ts_server/

It looks like Fabrice evaluates the models using something called LM Evaluation Harness:

https://github.com/EleutherAI/lm-evaluation-harness

I have no idea what this is yet, but would be nice to study it and try to integrate it here and in other ggml-based projects. This will be very important step needed to estimate the quality of the generated output and see if we are on the right track.

Contributor guide

Research direction
Review the LM Evaluation Harness documentation and source code to understand how it evaluates language models. Identify the key interfaces and data formats, then explore how to adapt llama.cpp's inference to be compatible. Propose a design for integration, possibly by wrapping llama.cpp's API in a Python interface or implementing a C++ harness.
Tech stack
python
Domain
testingperformancemachine learning
Issue type
Research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
C++PythonLM Evaluation Harness basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30