EvalPort export/import helpers for bridging evaluate's aggregate-only API to portable per-example results

Abierto
  • ¿Algo no está bien?
#800 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
machine-learning

Línea de trabajo

Empieza leyendo EvaluationModule.compute y el uso público de evaluate.load y metric.add_batch descrito en el issue. Compara la implementación de referencia probada en EvalPort's adapters/huggingface-evaluate-openeval-adapter con las convenciones de evaluate, incluida su suite de 25 tests. Se considera terminado cuando exista una API pública acordada que preserve el agregado por batch, realice llamadas reales a compute por ejemplo, documente las limitaciones a nivel de corpus y tenga tests upstream.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Motivation

I maintain EvalPort, an open, framework-agnostic JSON format for LLM evaluation test cases, graders, and results (schema-validated Suite/ResultSet documents — think "a portable file format eval tools can read/write instead of everyone inventing their own"). It already has independently-tested adapter packages for LangSmith, MLflow, Ragas, Braintrust, Arize Phoenix, Weave, and 16 other frameworks.

While building the evaluate one, I ran into a real, specific friction point that isn't a bug in evaluate — it's a genuine shape mismatch worth flagging directly. EvaluationModule.compute(predictions=..., references=...) always returns a whole-batch aggregate (confirmed against 0.4.6: exact_match, accuracy, f1, bleu all behave this way), which is exactly right for evaluate's own purpose. But anyone trying to export evaluate results into a per-example format (for storing in a dataset, diffing two model versions example-by-example, or feeding a downstream eval-tracking tool) has no supported path to get there — you either write your own per-example loop yourself, or you're stuck with only the aggregate.

I already solved this on my side (real per-example .compute() calls, one call per example, never interpolated from the aggregate — no fabricated numbers) and it works well enough that I'd like to check whether something like it belongs upstream, since I'd guess I'm not the only person who's hit this.

Feature description

A small, optional helper — something like evaluate.compute_per_example(metric, predictions, references, **kwargs) — that returns both the real whole-batch aggregate evaluate already computes today and a list of real per-example scores, obtained by calling the same metric's .compute() once per individual (prediction, reference) pair rather than once for the whole batch. No new metric logic, no new dependency — just a thin convenience wrapper around the existing .compute() call pattern, with a clear docstring caveat that per-example scoring isn't a meaningful concept for corpus-level statistics (classic BLEU being the standard example) the way it is for exact_match/accuracy/per-item f1.

If this is something the maintainers would want as a PR, I'm happy to open one against the API shape you'd prefer (a module-level function, a method on EvaluationModule, or something else) — I already have a tested reference implementation (compute_per_example) I built and validated for the standalone EvalPort adapter that I'd adapt to fit evaluate's own conventions rather than paste in as-is.

Code snippet demonstrating current vs. desired use
import evaluate

metric = evaluate.load("exact_match")
predictions = ["Paris", "5", "Berlin"]
references = ["Paris", "4", "Berlin"]

# Today: only the aggregate is available from the public API
metric.add_batch(predictions=predictions, references=references)
result = metric.compute()
# {"exact_match": 0.666...}  -- which examples failed? not recoverable from this alone

# Desired: an opt-in helper that also returns real per-example scores
aggregate, per_example = evaluate.compute_per_example("exact_match", predictions, references)
# aggregate == {"exact_match": 0.666...}          <- same real batch call as today
# per_example == [1.0, 0.0, 1.0]                  <- one real .compute() call per example, not interpolated
Additional context

For anyone curious what this looks like end-to-end (including the honesty caveat about corpus-level metrics), the standalone EvalPort adapter is public and tested: huggingface-evaluate-openeval-adapter (25/25 tests passing against the real installed evaluate package, README has a full "why this looks different" writeup). Not asking for an EvalPort integration itself — just flagging a real API gap I hit and offering to help close it if it's wanted upstream. Totally fine if this isn't a direction the maintainers want to take evaluate in; the adapter works standalone either way.

Lenguaje dominante
Python
Estrellas
2.5k
Forks
368
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de huggingface/evaluate

Todos los issues de huggingface/evaluate

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.