Coverage lines on pull requests are incorrect when not branched off latest commit of base branch

Open
#961 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
git, github, github-actions
Domain
ci-cd, testing-qa

Research direction

Reproduce the behavior in the linked codecov-bug-demo repository by creating a pull request from a non-latest base commit and running a pull_request workflow with actions/checkout. Compare the merge commit's coverage lines with the PR branch files in the Codecov website; done means the mismatch is explained and a concrete product-side change is identified.

Written by the indexing model from the issue text.

Description

Describe the bug

When a GitHub Actions workflow runs on the pull_request event, actions/checkout by default does not check out the pull request branch itself, but merges it into the base branch and checks out this merge commit.

This means that the reported coverage is for this merge commit, but the Codecov website does not account for this. When you view the coverage for a PR on the Codecov website, it shows the files as they are in the PR branch, meaning the coverage lines are incorrect (when there have been other changes to the same files committed to the base branch).

Environment (please complete the following information):

  • Browser N/A
  • System N/A

To Reproduce

This is what I did in a simple test repo: https://github.com/MetRonnie/codecov-bug-demo

  1. Create a pull request off a commit that is not the latest commit on the base branch

Expected behavior
I guess the only solution is for the Codecov website to display the same internal merge commit that actions/checkout checked out, rather than the head commit of the PR

Screenshots

Executable lines are shown as unexecutable and vice versa!

Image

Here is the commit graph.

Image

The coverage hits/misses match the lines of the latest commit on master (while the displayed line contents do not match)

Image

Additional context

Originally reported 2 years ago in https://github.com/codecov/feedback/issues/556 but has not been addressed. A colleague ran into this today and was thoroughly confused.

Dominant language
No language data
Stars
52
Forks
25
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from codecov/feedback

All issues in codecov/feedback

Similar issues

More DevOps issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.