MultipleChoice crashes when optional instructions is None

Open Beginner friendly
#1,775 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python
Domain
testing-qa

Research direction

Start in evals/elsuite/multiple_choice.py at MultipleChoice.init(), then trace how the prompt is built during evaluation. Add the requested regression test with instructions=None and verify that evaluation completes, while explicit non-empty instructions remain unchanged.

Written by the indexing model from the issue text.

Description

Describe the bug

evals.elsuite.multiple_choice.MultipleChoice.__init__() annotates instructions as optional:

instructions: Optional[str] = ""

but stores the value unchanged and later builds the prompt with:

self.instructions + "\nPlease answer with the letter of the correct answer."

Passing instructions=None, which is valid according to the public type, therefore raises TypeError during evaluation.

Expected behavior

None should behave like the existing default empty instruction string. Explicit non-empty instruction text should remain unchanged.

Proposed fix

Normalize None to "" in the constructor and add a regression test showing that a sample can be evaluated when instructions=None.

Dominant language
Python
Stars
19.5k
Forks
3.1k
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openai/evals

All issues in openai/evals

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.