docs out of date
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 68/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- python
- Domain
- documentation
Research direction
Start with docs/custom-eval.md and compare its sample against the current evals code and commit 64fb72a9a83b5d14dc765dbd8722f2bbbab5ef76. Run the documented example with Python 3.10 to verify it no longer references check_sampled_text and that its sample structure matches the current format. Done means the documentation runs successfully and reflects the current API.
Written by the indexing model from the issue text.
Description
Describe the bug
cods at https://github.com/openai/evals/blob/main/docs/custom-eval.md are out of date:
- the sample in the loop is an array so instead of
{"role": "system", "content": sample["problem"], "name": "example_user"},
{"role": "system", "content": sample["answer"], "name": "example_assistant"},
]
else:
prompt += [{"role": "user", "content": sample["problem"]}]```
the code should be:
prompt += [
{"role": "system", "content": sample[0]["content"], "name": "example_user"},
{"role": "system", "content": sample[1]["content"], "name": "example_assistant"},
]
else:
prompt += [{"role": "user", "content": sample[0]["content"]}]
- the check_sampled_text function doesn't exist after https://github.com/openai/evals/commit/64fb72a9a83b5d14dc765dbd8722f2bbbab5ef76
To Reproduce
- try running code in https://github.com/openai/evals/blob/main/docs/custom-eval.md
Code snippets
No response
OS
macOs
Python version
Python 3.10
Library version
1.0.3.post1
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from openai/evals
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100