opensearch-project/OpenSearch

[Gradle Check Retry] Publish/Record tests which pass after a retry to address flakiness

Open

#2,797 opened on Apr 6, 2022

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Java (1,505 forks)batch import
Build Libraries & Interfacesenhancementgood first issuehacktoberfest

Repository metrics

Stars
 (8,123 stars)
PR merge metrics
 (Avg merge 5d 9h) (266 merged PRs in 30d)

Description

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

  • As a part of the solution, there should be a record of logs and/or a comment on the PR in case of test failures
  • The generated report has a log of the tests that fail even though the overall job succeeds
  • Record the frequent offenders to address and mitigate the underlying root cause for flakiness

Describe alternatives you've considered

  • Manually keeping track of failed tests from the gradle check bot reports

Additional context

  • N/A

Contributor guide

Research direction
Investigate the retry mechanism from PR #2638, understand how test failures are currently handled, and design a method to log and report flaky tests that pass after retry.
Tech stack
java
Domain
backendtesting qa
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
GitJavaGradle
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40