[BUG] TruthfulQA MC2 drops true answers after a gap in gold indexes

Open Beginner friendly
#1,442 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@salma2vec is already working on this.

Since Oct 20, 2026.

  • #1444 by @salma2vec — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
testing-qa

Research direction

Start in lighteval/metrics/harness_compatibility/truthful_qa.py at TruthfulqaMCMetrics, then inspect its tests for MC2 scoring and gold-index handling. Reproduce with the supplied Doc and ModelResponse, and compare both answer orders. Done when MC2 returns 0.7 for both orders while MC1 remains true.

Written by the indexing model from the issue text.

Description

Describe the bug

TruthfulqaMCMetrics stops identifying correct MC2 answers at the first gap in the gold indexes. Later true answers are counted as false, although the task adapter preserves the supplied answer order and derives gold indexes from the labels.

To Reproduce

import math
from lighteval.metrics.harness_compatibility.truthful_qa import TruthfulqaMCMetrics
from lighteval.models.model_output import ModelResponse
from lighteval.tasks.requests import Doc

metric = TruthfulqaMCMetrics()
doc = Doc(
    query="Question",
    choices=["correct MC1", "wrong MC1", "true A", "false B", "true C"],
    gold_index=[0, 2, 4],
    specific={"len_mc1": 2},
)
response = ModelResponse(logprobs=[math.log(p) for p in [.9, .1, .2, .3, .5]])
print(metric.compute(doc, response))

Actual: MC1 is true; MC2 is 0.2. Moving true C and its score before false B, and updating the gold indexes, changes MC2 to 0.7.

Expected behavior

MC2 should be 0.7 in both orders: normalized probability mass assigned to all true answers, as in TruthfulQA's reference computation. Partitioning MC2 scores by the actual gold indexes would preserve the existing MC1 and contiguous-label behavior.

This demonstrates a valid-input/order-invariance problem. It does not establish a score change on the current default dataset; the historical dataset-row example in the source comment was not reproducible in a check of current rows 300–399.

Version info

Main commit 7fa19dc6f53a713ba8bb6541e7ed66dad2e24fd7, lighteval 0.13.1.dev0, Python 3.12 on macOS. Offline reproduction with the declared development dependencies.

Dominant language
Python
Stars
2.6k
Forks
612
Avg merge
2d 3h
Merged PRs (30d)
3

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/lighteval

All issues in huggingface/lighteval

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.