Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

BUG QuestionAnswerScorer counts a wrong option as correct when the answer appears inside a longer word or number

Open
  • Is something wrong?
#3,158 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@barkha1research-ai is already working on this.

Since Oct 12, 2026.

  • #3159 by @barkha1research-ai — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
python
Domain
ai, testing

Research direction

Start by reading QuestionAnswerScorer and the AnswerMatches matching logic; reproduce the reported case with the provided scorer snippet. Check the scorer tests for substring matching and add coverage for answers embedded in longer words or numbers. Done means the correct option still scores True while the wrong option containing it scores False; pull request #3159 is already open for this issue.

Written by the indexing model from the issue text.

Description

Bug: triage
Describe the bug

QuestionAnswerScorer checks its patterns with a plain substring test (text.lower() in response.lower()), so the correct answer also matches when it is only part of a longer word or number. A response that picks a wrong option is then scored as correct.

For example, with the correct answer 1, the response Option 2: 1000 scores True because 1000 contains 1. The same happens with 10000 inside 1000000, Vitamin B inside Vitamin B12, and the label pattern 1: inside 11:.

I ran the scorer over the WMDP test set (3,668 questions, the dataset used in the benchmark docs), answering each wrong option in the benchmark's Option {index}: {choice} format. 183 wrong answers, across 135 questions, are scored as correct.

Steps/Code to Reproduce
import asyncio

from pyrit.models import AnswerMatches, ContentScorable, ScoringExpectation
from pyrit.score import QuestionAnswerScorer
from pyrit.setup import IN_MEMORY, initialize_pyrit_async


async def main():
    await initialize_pyrit_async(memory_db_type=IN_MEMORY)
    scorer = QuestionAnswerScorer()
    expectation = ScoringExpectation(
        conditions=[
            AnswerMatches(
                correct_answer="1",
                correct_answer_label="1",
            )
        ]
    )

    for response in ["Option 1: 1", "Option 2: 1000"]:
        scores = await scorer.score_async(
            scorable=ContentScorable(value=response),
            expectation=expectation,
        )
        print(f"{response!r}: {scores[0].get_value()}")


asyncio.run(main())
Expected Results
'Option 1: 1': True
'Option 2: 1000': False
Actual Results
'Option 1: 1': True
'Option 2: 1000': True
Versions
  • PyRIT version: installed from main (93bf08e4) in editable mode
  • Python version: 3.12

I have a fix that only matches the answer when it is not part of a longer word, and would like to open a PR for it.

Dominant language
Python
Stars
4.6k
Forks
944
Avg merge
2d 20h
Merged PRs (30d)
309

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/PyRIT

All issues in microsoft/PyRIT

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.