[BUG] TruthfulQA MC2 drops true answers after a gap in gold indexes
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- testing-qa
Research direction
Start in lighteval/metrics/harness_compatibility/truthful_qa.py at TruthfulqaMCMetrics, then inspect its tests for MC2 scoring and gold-index handling. Reproduce with the supplied Doc and ModelResponse, and compare both answer orders. Done when MC2 returns 0.7 for both orders while MC1 remains true.
Written by the indexing model from the issue text.
Description
Describe the bug
TruthfulqaMCMetrics stops identifying correct MC2 answers at the first gap in the gold indexes. Later true answers are counted as false, although the task adapter preserves the supplied answer order and derives gold indexes from the labels.
To Reproduce
import math
from lighteval.metrics.harness_compatibility.truthful_qa import TruthfulqaMCMetrics
from lighteval.models.model_output import ModelResponse
from lighteval.tasks.requests import Doc
metric = TruthfulqaMCMetrics()
doc = Doc(
query="Question",
choices=["correct MC1", "wrong MC1", "true A", "false B", "true C"],
gold_index=[0, 2, 4],
specific={"len_mc1": 2},
)
response = ModelResponse(logprobs=[math.log(p) for p in [.9, .1, .2, .3, .5]])
print(metric.compute(doc, response))
Actual: MC1 is true; MC2 is 0.2. Moving true C and its score before false B, and updating the gold indexes, changes MC2 to 0.7.
Expected behavior
MC2 should be 0.7 in both orders: normalized probability mass assigned to all true answers, as in TruthfulQA's reference computation. Partitioning MC2 scores by the actual gold indexes would preserve the existing MC1 and contiguous-label behavior.
This demonstrates a valid-input/order-invariance problem. It does not establish a score change on the current default dataset; the historical dataset-row example in the source comment was not reproducible in a check of current rows 300–399.
Version info
Main commit 7fa19dc6f53a713ba8bb6541e7ed66dad2e24fd7, lighteval 0.13.1.dev0, Python 3.12 on macOS. Offline reproduction with the declared development dependencies.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 612
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 3
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/lighteval
-
[BUG] arc_agi_2: a correct grid in the format the prompt asks for scores 0 (exact_match gold is a one-line list)Possibly taken @BahaaMattar claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
huggingface/lighteval#1510 ·
Maintainers usually reply within 1 day
-
[BUG] vllm_main_tests.yaml: continue-on-error hides a canary 23 scheduled runs deep in failurePossibly taken @voropaevv claimed this 9 days ago. Open
Difficulty 2/5 Under an hour Newbie friendliness 76/100
huggingface/lighteval#1486 ·
Maintainers usually reply within 1 day
-
[BUG] Belebele cf tasks fail for Latvian, Lithuanian, Maltese and Slovenian: missing translation literalsPossibly taken @inchang-ing claimed this 15 days ago. Openbug
Difficulty 1/5 1-3 hours Newbie friendliness 74/100
huggingface/lighteval#1471 ·
Maintainers usually reply within 1 day
-
[BUG] Stderr of `_average` and `all` rows is the mean of the subtask stderrs (about 7x too large for MMLU)Possibly taken @Andreas-Hum claimed this 17 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
huggingface/lighteval#1466 ·
Maintainers usually reply within 1 day
-
[BUG] Reasoning tag removal fails when <think> is part of the input promptPossibly taken @selimfedakar claimed this 21 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
huggingface/lighteval#1453 · 1 comment ·
Maintainers usually reply within 1 day
All issues in huggingface/lighteval
Similar issues
-
integration: media_player
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
home-assistant/core#193344 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
daniel-ospina/tortoise#13607 ·
Maintainers usually reply within 1 day
-
documentation priority: medium
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day