Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Resolve double counting across original and multilingual evals

Ouverte
  • Quelque chose ne va pas ?
#2 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
25/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
javascript
Domaine
analytics

Piste de recherche

Start by reading the existing canonical eval/scoring-group and aggregation logic, plus the warnings/inspection views and configuration documentation; no file paths are named in the issue. First agree on the grouping and overlap policy, then trace how it affects standard and English-balanced scores and effective eval weights. Done means implementing and documenting the policy, exposing protocol/task identity and unused data, and passing the listed representative A/B, coverage, reconciliation, and aggregation checks.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Resolve how original and multilingual eval variants contribute to the same comparison, including repeated English measurements. Follow-up to #1; this work is deferred from that PR and does not block its merge.

Status: open follow-up. Ownership: Quickdash maintainer (unassigned). Next action: agree on scoring groups and language exclusions, then implement and test the policy. No implementation dependency on an external service.

Current behavior and evidence:

  • MGSM combines global_mgsm and mgsm_native_cot. Both can contribute English; the inspected export also has German, Spanish, and French in both protocols. The English entries use different few-shot settings (0 versus 5).
  • PIQA combines piqa and global_piqa_completions. Both can contribute English, with different few-shot settings (10 versus 0) and sample counts.
  • ARC Challenge combines arc_challenge and arc_challenge_mt. The inspected export has English only in the original task, so there is no repeated English measurement there. Other exports still need a defined policy.
  • MMLU and Global MMLU are intentionally separate evals and both include English. Review overlap without silently undoing that decision.

These observations establish repeated language measurements, not identical underlying question sets. Investigate task implementations/data before claiming item-level duplication. No private scores or model identities are needed in this issue.

Sampo proposed keeping translated variants separately named and dropping English from multilingual results where the original supplies English. Decide whether this is the desired policy and how to handle overlapping non-English MGSM variants and missing original-language coverage.

The dashboard already has two naming levels: a canonical eval/scoring group, and exact source task names visible in expanded views. There is no intermediate protocol subgroup. Splitting canonical evals changes their equal shares within a category; it is not merely a label change. English-balanced modes retain the configured English share but average multiple English measurements within it.

Acceptance checks:

  • Document the chosen grouping and overlap policy, including original/multilingual naming, English exclusion, non-English overlap, and MMLU/Global MMLU treatment.
  • Keep global interpretation rules distinct from comparison membership; express exclusions explicitly and make unused data visible through the existing warnings/inspection views.
  • Verify the policy against representative A/B inputs, including multilingual-only data and asymmetric coverage. Missing measurements must not silently enter one side's score.
  • Test standard, English-per-eval, and English-per-category aggregation, contribution reconciliation, and any intended change to effective eval weights.
  • Ensure breakdowns expose protocol/task identity, and update configuration documentation with a concrete example.
  • Explain remaining differences from Sampo's aggregation once his grouping policy is established; matching metric/baseline tables alone is insufficient.
Langage dominant
Python
Étoiles
0
Forks
1
Merge moyen
2 h
PR mergées (30 j)
14

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.