[YSQL] Migrate custom YBTestRunner subclasses to annotation-based test filtering
#31675 opened on May 15, 2026
Description
Jira Link: DB-21420
Description
Summary
We have a proliferation of YBTestRunner subclasses whose only purpose is to override shouldRunTests() with a build-type or platform check. Each new condition requires a new class (and a parallel class for parameterized tests). #31674 introduces a @RequiresReleaseBuild annotation checked in YBTestRunner.getChildren() as a first instance of this pattern. This issue tracks generalizing that approach to replace the existing custom runner subclasses.
Current state
Non-parameterized runners (extend YBTestRunner)
| Runner class | Condition | Approximate usage count |
|---|---|---|
YBTestRunnerNonTsanOnly |
!isTSAN() |
~56 test classes |
YBTestRunnerNonTsanAsan |
!isTSAN() && !isASAN() |
base class for others |
YBTestRunnerNonSanitizersOrMac |
!isSanitizer() && IS_LINUX |
several test classes |
YBTestRunnerNonMac |
IS_LINUX |
several test classes |
YBTestRunnerNonSanOrAArch64Mac |
!isSanitizer() && !IS_AARCH64 && IS_LINUX |
several test classes |
YBTestRunnerYsqlConnMgr |
IS_LINUX |
~35 test classes |
Parameterized runners (extend YBParameterizedTestRunner)
| Runner class | Condition |
|---|---|
YBParameterizedTestRunnerNonTsanOnly |
!isTSAN() |
YBParameterizedTestRunnerNonTsanAsan |
!isTSAN() && !isASAN() |
YBParameterizedTestRunnerNonSanitizersOrMac |
!isSanitizer() && IS_LINUX |
YBParameterizedTestRunnerYsqlConnMgr |
IS_LINUX |
Problems with the runner approach
- Combinatorial explosion: Each new condition (build type, platform, feature flag) requires a new runner class — and a parallel one for parameterized tests.
- Class-level granularity only: If one method in a class needs a different condition, you must either split the class or fall back to
assumeTrue(which wastes setUp time — the problem that motivated #31674). - No composition: Combining conditions means creating yet another subclass (e.g.,
YBTestRunnerNonSanOrAArch64Mac).
Proposed approach
Generalize the @RequiresReleaseBuild pattern (introduced in #31674) into a flexible annotation framework checked by the base runners (YBTestRunner and YBParameterizedTestRunner).
Design sketch
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface SkipTestOn {
String[] buildTypes() default {}; // e.g. {"tsan", "asan", "fastdebug"}
boolean macOnly() default false; // skip when IS_LINUX
boolean linuxOnly() default false; // skip when !IS_LINUX
boolean aarch64() default false; // skip on aarch64
String reason() default ""; // for documentation
}
Or keep it as a family of simple, self-documenting annotations:
@RequiresReleaseBuild(introduced in #31674)@SkipOnTSAN@SkipOnSanitizers@RequiresLinux
Both YBTestRunner.getChildren() and YBParameterizedTestRunner.getChildren() would filter based on these annotations. Type-level annotations would apply to all methods in the class.
Migration path
- Phase 1: Implement the new annotations and filtering in both
YBTestRunnerandYBParameterizedTestRunner.@RequiresReleaseBuildfrom #31674 serves as the template. - Phase 2: Migrate test classes from custom runners to
@RunWith(YBTestRunner.class)+ annotations, one runner at a time (e.g., start withYBTestRunnerNonTsanOnlywhich has the most usages). - Phase 3: Delete the now-unused runner subclasses.
Each phase can be done incrementally. Annotations and runners can coexist during migration.
Benefits
- No new classes for new conditions — just add an annotation.
- Method-level granularity — no need to split test classes or use
assumeTrue. - Composable — multiple annotations on one method/class combine naturally.
- Self-documenting — the skip reason lives right next to the test.
- Works for parameterized tests too — same annotations, no parallel class hierarchy.
Issue Type
kind/failing-test
Warning: Please confirm that this issue does not contain any sensitive information
- I confirm this issue does not contain any sensitive information.