Bench: pin the LiveCodeBench dataset revision and check the cached file
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
Research direction
Start with atlas/bench/datasets/livecodebench.py, especially download and _parse, and trace how the dataset is fetched, cached, and reported. Add tests around the loader, including a changed-cache case. Done means the loader pins a revision and hash, rejects or refetches a mismatched cache with a clear message, and the benchmark report names the revision.
Written by the indexing model from the issue text.
Description
What needs doing
Fetch one fixed revision of the dataset. Keep its hash next to the
cache. Fetch again, or stop, when the cached file does not match.
Why
Benchmark numbers must be repeatable. Today the rows come from the
newest upstream state, and the cache is loaded as it is.
Done when
- Revision and hash are constants in the loader.
- A test with a changed cache file stops with a clear message.
- The benchmark report names the dataset revision.
Where to start
atlas/bench/datasets/livecodebench.py(download,_parse).
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 197
- Avg merge
- 14h 1m
- Merged PRs (30d)
- 201
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from inferstep/ATLAS
-
Bootstrap: the last two steps run as the user who owns the installMay be free again @assassinationss claimed this 16 days ago, and no pull request is open. Openarea/install reliability
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
inferstep/ATLAS#412 · 4 comments ·
Maintainers usually reply within 1 day
-
area/cli
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
area/cli area/lens status/ready
Difficulty 2/5 Half a day Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
area/ci status/ready
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
inferstep/ATLAS#245 · 4 comments ·
Maintainers usually reply within 1 day
-
area/docs documentation help wanted
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Maintainers usually reply within 1 day
Similar issues
-
README.zh-CN/ja/es still hard-code 501 test counts after README.md went count-freePossibly taken @Aaqibhafeezkhan claimed this today. Openbeginner documentation good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
BistaDinesh03/critique#164 · 1 comment ·
Maintainers usually reply within 1 day
-
bug IO
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
MrNeRF/LichtFeld-Studio#3824 ·
Maintainers usually reply within 1 day
-
bug confirmed issue
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
open-webui/open-webui#33200 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
-
The changelog workflow excludes otelbot but not dependabot, so every Dependabot pull request fails the changelog checkPossibly taken @ocelotl claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 82/100
open-telemetry/opentelemetry-python#5825 ·
Maintainers usually reply within 2 days