MakeMeSay get_content fails on its documented dict response type
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 84/100
Research direction
Start with evals.elsuite.make_me_say.utils.get_content() and inspect its existing handling of CompletionResult and attribute-style responses. Add focused regression coverage for dictionary and CompletionResult inputs, while confirming attribute-style behavior remains unchanged; done means the documented dictionary returns "hello" without breaking the existing cases.
Written by the indexing model from the issue text.
Description
Describe the bug
evals.elsuite.make_me_say.utils.get_content() is annotated to accept either a dict or a CompletionResult:
def get_content(response: Union[dict, CompletionResult]) -> str:
It handles CompletionResult through get_completions(), but the fallback assumes attribute-style OpenAI response objects:
return response.choices[0].message.content
A normal dictionary response such as:
{"choices": [{"message": {"content": "hello"}}]}
therefore raises AttributeError even though dict is explicitly part of the public input type.
Expected behavior
Dictionary responses should return response["choices"][0]["message"]["content"]; existing CompletionResult and attribute-style response behavior should remain unchanged.
Proposed fix
Add an explicit dictionary branch and focused regression coverage for dict and CompletionResult inputs.
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from openai/evals
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100