[BUG] never-worked baseline handling is not fully integrated into trusted CI reporting
Maintainers usually reply within 6 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
Research direction
Start with the controller and regression tests covering trusted baseline refresh, PR baseline mutation, platform-specific never-worked reporting, and PR comments. Trace how BaselineStatus, baseline history, migration backfill, and regression-result logic flow through CI reporting. Done means trusted commit runs refresh baselines, PR and fork runs do not, and comments distinguish new, existing, and platform-specific never-worked failures.
Written by the indexing model from the issue text.
Description
Sample platform commit (found at the bottom of each page) : 66f057ab
In raising this issue, I confirm the following (please check boxes, eg [X]):
- I have read and understood the contributors guide.
- I have checked that the bug-fix I am reporting can be replicated, or that the feature I am suggesting isn't already present.
- I have checked that the issue I'm posting isn't already reported.
- I have checked that the issue I'm posting isn't already solved and no duplicates exist in closed issues and in opened issues
- I have checked the pull requests tab for existing solutions/implementations to my issue/suggestion.
My familiarity with the project is as follows (check one, eg [X]):
- I have never visited/used the platform.
- I have used the platform just a couple of times.
- I have used the platform extensively, but have not contributed previously.
- I am an active contributor to the platform.
Summary
The recently added BaselineStatus model state still needs end-to-end integration in the CI and reporting flow.
Today the platform only partially understands never_worked:
- baseline state can exist at the model and schema level, but reporting can still misclassify failures
- baseline history should only be refreshed from trusted main-repo commit runs, not from unmerged PR or fork runs
- never worked needs to be evaluated per platform using last_passed_on_linux and last_passed_on_windows, otherwise a test that passed on Linux can be misreported on Windows
- historical migration backfill should trust known pass history and avoid over-classifying ambiguous old failures
Expected behavior
- completed main-repo commit runs refresh baseline status using full regression-result logic
- PR and fork runs do not mutate shared baseline history
- PR comments distinguish new failures, failures already failing on master, and tests that have never worked on the current platform
- migration backfill marks tests as established only when trusted pass history exists; otherwise they remain unknown until refreshed by trusted runs
Why this matters
This keeps CI failure signals accurate. Without it, contributors can get misleading PR comments and unmerged runs can pollute the baseline used to classify regressions.
Verification
Focused controller and regression tests cover trusted baseline refresh, no PR baseline mutation, platform-specific never-worked reporting, and existing PR comment behavior.
- Dominant language
- Python
- Stars
- 30
- Forks
- 85
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 21
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from CCExtractor/sample-platform
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
CCExtractor/sample-platform#1194 · 1 comment ·
Maintainers usually reply within 6 days
-
Difficulty 3/5 1-2 days Newbie friendliness 84/100
CCExtractor/sample-platform#1192 ·
Maintainers usually reply within 6 days
-
Difficulty 3/5 1-2 days Newbie friendliness 42/100
CCExtractor/sample-platform#1082 ·
Maintainers usually reply within 6 days
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
CCExtractor/sample-platform#1079 ·
Maintainers usually reply within 6 days
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
CCExtractor/sample-platform#1062 · 2 comments ·
Maintainers usually reply within 6 days
All issues in CCExtractor/sample-platform
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
PedestrianDynamics/pyFDS-Evac#343 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
theskumar/python-dotenv#708 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 2 days
-
Docs Timedelta
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pandas-dev/pandas#69919 ·
Maintainers usually reply within 1 day
-
API documentation
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
zephyrproject-rtos/west#1009 · 2 comments ·
Maintainers usually reply within 3 days