Split benchmarks into warm and cold benchmarks
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 45/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- performance, testing
Research direction
The issue does not name benchmark files or tests, so start by locating the benchmark definitions that exercise caching behavior. Separate measurements for first-run and repeated-call behavior, then run the affected benchmark suite and verify that warm and cold results are reported independently.
Written by the indexing model from the issue text.
Description
Some of the benchmarks are testing functions which behave differently on the first run because they cache data that they then use in repeated calls.
We should split the benchmark measuring such functions into warm and cold benchmarks. Otherwise the benchmarks have a large spread between the max value (cold run) and the min value and the mean becomes less meaningful.
- Dominant language
- Python
- Stars
- 531
- Forks
- 147
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from graphql-python/graphql-core
-
Difficulty 4/5 3-5 days Newbie friendliness 50/100
graphql-python/graphql-core#272 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
graphql-python/graphql-core#269 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
graphql-python/graphql-core#267 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
graphql-python/graphql-core#247 · 8 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
graphql-python/graphql-core#223 · 1 comment ·
All issues in graphql-python/graphql-core
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
anthropics/skills#1811 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
speaches-ai/speaches#678 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
datalayer/mcp-compose#42 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
conda-forge/spacy-feedstock#177 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
UKGovernmentBEIS/inspect_evals#2523 ·