Coverage lines on pull requests are incorrect when not branched off latest commit of base branch
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Active
- Tech stack
- git, github, github-actions
- Domain
- ci-cd, testing-qa
Research direction
Reproduce the behavior in the linked codecov-bug-demo repository by creating a pull request from a non-latest base commit and running a pull_request workflow with actions/checkout. Compare the merge commit's coverage lines with the PR branch files in the Codecov website; done means the mismatch is explained and a concrete product-side change is identified.
Written by the indexing model from the issue text.
Description
Describe the bug
When a GitHub Actions workflow runs on the pull_request event, actions/checkout by default does not check out the pull request branch itself, but merges it into the base branch and checks out this merge commit.
This means that the reported coverage is for this merge commit, but the Codecov website does not account for this. When you view the coverage for a PR on the Codecov website, it shows the files as they are in the PR branch, meaning the coverage lines are incorrect (when there have been other changes to the same files committed to the base branch).
Environment (please complete the following information):
- Browser N/A
- System N/A
To Reproduce
This is what I did in a simple test repo: https://github.com/MetRonnie/codecov-bug-demo
- Create a pull request off a commit that is not the latest commit on the base branch
Expected behavior
I guess the only solution is for the Codecov website to display the same internal merge commit that actions/checkout checked out, rather than the head commit of the PR
Screenshots
Executable lines are shown as unexecutable and vice versa!
Here is the commit graph.
The coverage hits/misses match the lines of the latest commit on master (while the displayed line contents do not match)
Additional context
Originally reported 2 years ago in https://github.com/codecov/feedback/issues/556 but has not been addressed. A colleague ran into this today and was thoroughly confused.
- Dominant language
- No language data
- Stars
- 52
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from codecov/feedback
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
-
Bundle analysis: `normalizePath` replaces everything after the first `-` in a file name, so different assets collidePossibly taken @ChiptenIsCool claimed this 16 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Docs Waiting for: Product Owner
Difficulty 1/5 Under an hour Newbie friendliness 65/100
-
Difficulty 4/5 3-5 days Newbie friendliness 52/100
-
Difficulty 4/5 1-2 days Newbie friendliness 42/100
All issues in codecov/feedback
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
nutmos/expensetrackr#157 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
janitor-team/janitor#2613 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
keyxmakerx/Chronicle#2550 ·
Maintainers usually reply within 1 day
-
ci/cd javascript medium sdk
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
bug good first issue help wanted rust type:infrastructure
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Open-Glassbeetle/glassbeetle#374 ·
Maintainers usually reply within 1 day