Benchmarks can be sensitive to different resolutions
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python
- Domain
- performance, tooling
Research direction
Start by reviewing how the benchmark tools currently record dependency resolutions and compare the outputs across tools. Define the configuration format for approved differences and make the script flag unapproved differences; done means benchmark inputs can be checked for materially different resolutions.
Written by the indexing model from the issue text.
Description
There are more than one valid resolution to the dependencies being used, and as seen in #37, resolvers can choose different solutions. To minimize incorrectness in the benchmarks, the resolutions should ideally be identical in the end, e.g., so resolve benchmarks capture how long it takes to get to the same solution, and so install benchmarks are for the same set of files.
It'd be good to write a small script that compares the resolutions across the tools and flags differences. Differences should probably require approval in a configuration file to "allow". I think some small differences are okay, but if the tools have significantly different resolutions they can affect benchmarks in a way that's misleading.
- Dominant language
- Makefile
- Stars
- 50
- Forks
- 15
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lincolnloop/python-package-manager-shootout
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
lincolnloop/python-package-manager-shootout#37 · 3 comments · 1 reaction ·
-
Difficulty 4/5 3-5 days Newbie friendliness 50/100
-
Difficulty 4/5 3-5 days Newbie friendliness 28/100
lincolnloop/python-package-manager-shootout#5 · 4 comments · 1 reaction ·
All issues in lincolnloop/python-package-manager-shootout
Similar issues
-
area: cli area: dogs perf priority: P3
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
vllm-project/vllm#57936 · 1 comment ·
-
contributor: external needs review
Difficulty 2/5 1-3 hours Newbie friendliness 82/100