Provide three-state output: "changed", "not changed", "unsure"
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- rust
- Domain
- performance, testing-qa
Research direction
The issue names no files, tests, or entry points. Start by locating the Behrens-Fisher confidence classification and the benchmark result output, then determine how insufficient precision should be represented and tested. Done means Sightglass can distinguish changed, not changed, and unsure outcomes at the requested confidence.
Written by the indexing model from the issue text.
Description
Right now, Sightglass uses a single threshold based on a confidence interval computed by Behrens-Fisher to determine whether a sampled statistic shifted between configurations.
The result of this is that we get either "changed" (i.e., benchmark got 5% faster) or "not changed". However, the latter answer can also appear if we simply don't have enough data points to prove statistical significance, or if the system is too noisy.
This "false negative" is somewhat dangerous: we could make a change, see that it is performance-neutral according to Sightglass, and accept it, but actually we just didn't turn the knobs up high enough.
Ideally, Sightglass should provide a third output of "unsure" if the measurements aren't precise enough to prove either "changed" or "not changed" to the desired confidence.
- Dominant language
- WebAssembly
- Stars
- 88
- Forks
- 37
- Avg merge
- 5h 5m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from bytecodealliance/sightglass
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
bytecodealliance/sightglass#298 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
bytecodealliance/sightglass#297 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 38/100
bytecodealliance/sightglass#296 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
bytecodealliance/sightglass#275 · 1 comment ·
-
bytecodealliance/sightglass#267 · 1 comment · 1 assignee ·
All issues in bytecodealliance/sightglass
Similar issues
-
area: cli area: dogs perf priority: P3
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
vllm-project/vllm#57936 · 1 comment ·
-
contributor: external needs review
Difficulty 2/5 1-3 hours Newbie friendliness 82/100