yugabyte/yugabyte-db

[YSQL] Migrate custom YBTestRunner subclasses to annotation-based test filtering

Closed

#31675 opened on May 15, 2026

View on GitHub
 (0 comments) (0 reactions) (0 assignees)C (8,229 stars) (1,003 forks)batch import
area/ysqlhelp wantedkind/failing-testpriority/mediumstatus/awaiting-triage

Description

Jira Link: DB-21420

Description

Summary

We have a proliferation of YBTestRunner subclasses whose only purpose is to override shouldRunTests() with a build-type or platform check. Each new condition requires a new class (and a parallel class for parameterized tests). #31674 introduces a @RequiresReleaseBuild annotation checked in YBTestRunner.getChildren() as a first instance of this pattern. This issue tracks generalizing that approach to replace the existing custom runner subclasses.

Current state

Non-parameterized runners (extend YBTestRunner)

Runner class Condition Approximate usage count
YBTestRunnerNonTsanOnly !isTSAN() ~56 test classes
YBTestRunnerNonTsanAsan !isTSAN() && !isASAN() base class for others
YBTestRunnerNonSanitizersOrMac !isSanitizer() && IS_LINUX several test classes
YBTestRunnerNonMac IS_LINUX several test classes
YBTestRunnerNonSanOrAArch64Mac !isSanitizer() && !IS_AARCH64 && IS_LINUX several test classes
YBTestRunnerYsqlConnMgr IS_LINUX ~35 test classes

Parameterized runners (extend YBParameterizedTestRunner)

Runner class Condition
YBParameterizedTestRunnerNonTsanOnly !isTSAN()
YBParameterizedTestRunnerNonTsanAsan !isTSAN() && !isASAN()
YBParameterizedTestRunnerNonSanitizersOrMac !isSanitizer() && IS_LINUX
YBParameterizedTestRunnerYsqlConnMgr IS_LINUX

Problems with the runner approach

  1. Combinatorial explosion: Each new condition (build type, platform, feature flag) requires a new runner class — and a parallel one for parameterized tests.
  2. Class-level granularity only: If one method in a class needs a different condition, you must either split the class or fall back to assumeTrue (which wastes setUp time — the problem that motivated #31674).
  3. No composition: Combining conditions means creating yet another subclass (e.g., YBTestRunnerNonSanOrAArch64Mac).

Proposed approach

Generalize the @RequiresReleaseBuild pattern (introduced in #31674) into a flexible annotation framework checked by the base runners (YBTestRunner and YBParameterizedTestRunner).

Design sketch

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface SkipTestOn {
    String[] buildTypes() default {};   // e.g. {"tsan", "asan", "fastdebug"}
    boolean macOnly() default false;    // skip when IS_LINUX
    boolean linuxOnly() default false;  // skip when !IS_LINUX
    boolean aarch64() default false;    // skip on aarch64
    String reason() default "";         // for documentation
}

Or keep it as a family of simple, self-documenting annotations:

  • @RequiresReleaseBuild (introduced in #31674)
  • @SkipOnTSAN
  • @SkipOnSanitizers
  • @RequiresLinux

Both YBTestRunner.getChildren() and YBParameterizedTestRunner.getChildren() would filter based on these annotations. Type-level annotations would apply to all methods in the class.

Migration path

  1. Phase 1: Implement the new annotations and filtering in both YBTestRunner and YBParameterizedTestRunner. @RequiresReleaseBuild from #31674 serves as the template.
  2. Phase 2: Migrate test classes from custom runners to @RunWith(YBTestRunner.class) + annotations, one runner at a time (e.g., start with YBTestRunnerNonTsanOnly which has the most usages).
  3. Phase 3: Delete the now-unused runner subclasses.

Each phase can be done incrementally. Annotations and runners can coexist during migration.

Benefits

  • No new classes for new conditions — just add an annotation.
  • Method-level granularity — no need to split test classes or use assumeTrue.
  • Composable — multiple annotations on one method/class combine naturally.
  • Self-documenting — the skip reason lives right next to the test.
  • Works for parameterized tests too — same annotations, no parallel class hierarchy.

Issue Type

kind/failing-test

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.

Contributor guide