Coverage comment replaces "1 Tests Failed" with "All tests successful" when the failing test passed in another CI job

Open
  • Is something wrong?
#989 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Domain
testing

Research direction

Start by reading the linked investigation in get_pr_comment_agg, calc_test_id, get_test_status, and the notify task, then trace how JUnit results from multiple flagged uploads are combined. Check how the existing tests cover mixed pass and fail results across uploads. Done means the failure remains visible and the comment does not claim all tests passed when any upload reports a failure.

Written by the indexing model from the issue text.

Description

Describe the bug
We upload JUnit results from multiple CI matrix jobs, each with its own flag. A test has failed in one job but passed in others that upload later, Codecov first posted a "❌ 1 Tests Failed" comment but then replaced it with the coverage report, which even says "✅ All tests successful. No failed tests found."

See https://github.com/scrapy/scrapy/pull/8387

Screenshots

Image

Additional context
LLM-provided investigation:

get_pr_comment_agg (https://github.com/codecov/umbrella/blob/65af8ae7d13eae4e83f179e2937b1bf18de00ba9/apps/worker/services/test_analytics/ta_timeseries.py#L197-L207) keeps only the latest outcome of each test_id across all uploads of the commit: LAST(outcome, timestamp) … GROUP BY test_id. calc_test_id (https://github.com/codecov/umbrella/blob/65af8ae7d13eae4e83f179e2937b1bf18de00ba9/apps/worker/services/test_analytics/ta_timeseries.py#L46-L48) hashes only name,
classname and testsuite, not the flags. So a later pass in another job hides the failure, get_test_status (https://github.com/codecov/umbrella/blob/65af8ae7d13eae4e83f179e2937b1bf18de00ba9/apps/worker/helpers/ta_status.py) returns any_failures=False, all_passed=True, and the notify task (https://github.com/codecov/umbrella/blob/65af8ae7d13eae4e83f179e2937b1bf18de00ba9/apps/worker/tasks/notify.py#L229-L250) goes ahead and overwrites the comment.

Though, to be honest, I wouldn't expect it to say "All tests successful" when a test failed, even when not using flags.

Dominant language
No language data
Stars
52
Forks
26
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from codecov/feedback

All issues in codecov/feedback

Similar issues

More Testing & QA issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.