Eval grading: ignore or reject edits to a benchmark's own test files
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- devtools, testing-qa
Research direction
Start by reading the grading flow and how it receives a task's test files and configuration; the issue does not name specific files or tests. Compare the options of restoring the original files or rejecting changed tests, then use a toy task where the session edits tests to verify that the grade is unchanged.
Written by the indexing model from the issue text.
Description
What needs doing
Before grading, restore the benchmark's test files from the task (or
reject the run if they changed). An agent that edits or adds test
configuration (for example a planted conftest.py) must not change the
grade.
Why
This is a known way to pass tests without solving the task. Either arm
could hit it by accident.
Done when
- A toy task where the session edits the tests grades as if it had not.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 197
- Avg merge
- 16h 6m
- Merged PRs (30d)
- 168
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from inferstep/ATLAS
-
Bootstrap: the last two steps run as the user who owns the installMay be free again @assassinationss claimed this 11 days ago, and no pull request is open. Openarea/install reliability
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
inferstep/ATLAS#412 · 4 comments ·
Maintainers usually reply within 1 day
-
area/cli
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
area/cli area/lens status/ready
Difficulty 2/5 Half a day Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
area/ci status/ready
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
inferstep/ATLAS#245 · 4 comments ·
Maintainers usually reply within 1 day
-
area/sandbox security
Difficulty 3/5 1-2 days Newbie friendliness 62/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
eclipse-score/inc_someip_gateway#356 ·
Maintainers usually reply within 2 days
-
[BUG]: `GitHubRestAdapter.check_user_permission` and `has_write_access` raise `TypeError` (missing `params`), silently disabling the inactivity lifecyclePossibly taken @tarunagnihotri534 claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
AOSSIE-Org/Gitcord-GithubDiscordBot#167 · 1 comment ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
mihai-vlc/sublime-jsfmt#56 ·
-
comp/cli comp/plugins P2 type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
NousResearch/hermes-agent#141917 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
riscv/riscv-opcodes#441 ·