Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

EvalPort export/import helpers for bridging evaluate's aggregate-only API to portable per-example results

Aperta
  • Qualcosa non va?
#800 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
machine-learning

Direzione di ricerca

Inizia leggendo EvaluationModule.compute e l'uso pubblico di evaluate.load e metric.add_batch descritto nell'issue. Confronta l'implementazione di riferimento testata in EvalPort's adapters/huggingface-evaluate-openeval-adapter con le convenzioni di evaluate, inclusa la sua suite di 25 test. Il lavoro è completato quando esiste un'API pubblica concordata che preserva l'aggregato del batch, esegue chiamate reali a compute per ogni esempio, documenta le limitazioni a livello di corpus e include test upstream.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Motivation

I maintain EvalPort, an open, framework-agnostic JSON format for LLM evaluation test cases, graders, and results (schema-validated Suite/ResultSet documents — think "a portable file format eval tools can read/write instead of everyone inventing their own"). It already has independently-tested adapter packages for LangSmith, MLflow, Ragas, Braintrust, Arize Phoenix, Weave, and 16 other frameworks.

While building the evaluate one, I ran into a real, specific friction point that isn't a bug in evaluate — it's a genuine shape mismatch worth flagging directly. EvaluationModule.compute(predictions=..., references=...) always returns a whole-batch aggregate (confirmed against 0.4.6: exact_match, accuracy, f1, bleu all behave this way), which is exactly right for evaluate's own purpose. But anyone trying to export evaluate results into a per-example format (for storing in a dataset, diffing two model versions example-by-example, or feeding a downstream eval-tracking tool) has no supported path to get there — you either write your own per-example loop yourself, or you're stuck with only the aggregate.

I already solved this on my side (real per-example .compute() calls, one call per example, never interpolated from the aggregate — no fabricated numbers) and it works well enough that I'd like to check whether something like it belongs upstream, since I'd guess I'm not the only person who's hit this.

Feature description

A small, optional helper — something like evaluate.compute_per_example(metric, predictions, references, **kwargs) — that returns both the real whole-batch aggregate evaluate already computes today and a list of real per-example scores, obtained by calling the same metric's .compute() once per individual (prediction, reference) pair rather than once for the whole batch. No new metric logic, no new dependency — just a thin convenience wrapper around the existing .compute() call pattern, with a clear docstring caveat that per-example scoring isn't a meaningful concept for corpus-level statistics (classic BLEU being the standard example) the way it is for exact_match/accuracy/per-item f1.

If this is something the maintainers would want as a PR, I'm happy to open one against the API shape you'd prefer (a module-level function, a method on EvaluationModule, or something else) — I already have a tested reference implementation (compute_per_example) I built and validated for the standalone EvalPort adapter that I'd adapt to fit evaluate's own conventions rather than paste in as-is.

Code snippet demonstrating current vs. desired use
import evaluate

metric = evaluate.load("exact_match")
predictions = ["Paris", "5", "Berlin"]
references = ["Paris", "4", "Berlin"]

# Today: only the aggregate is available from the public API
metric.add_batch(predictions=predictions, references=references)
result = metric.compute()
# {"exact_match": 0.666...}  -- which examples failed? not recoverable from this alone

# Desired: an opt-in helper that also returns real per-example scores
aggregate, per_example = evaluate.compute_per_example("exact_match", predictions, references)
# aggregate == {"exact_match": 0.666...}          <- same real batch call as today
# per_example == [1.0, 0.0, 1.0]                  <- one real .compute() call per example, not interpolated
Additional context

For anyone curious what this looks like end-to-end (including the honesty caveat about corpus-level metrics), the standalone EvalPort adapter is public and tested: huggingface-evaluate-openeval-adapter (25/25 tests passing against the real installed evaluate package, README has a full "why this looks different" writeup). Not asking for an EvalPort integration itself — just flagging a real API gap I hit and offering to help close it if it's wanted upstream. Totally fine if this isn't a direction the maintainers want to take evaluate in; the adapter works standalone either way.

Lingua principale
Python
Stelle
2.5k
Fork
346
Merge medio
38m
PR unite (30g)
4

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di huggingface/evaluate

Tutte le issue di huggingface/evaluate

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.