enhancementhelp wantedquestion
Repository-Metriken
- Stars
- (537 Sterne)
- PR-Merge-Metriken
- (PR-Metriken ausstehend)
Beschreibung
As discussed in https://github.com/nodejs/TSC/issues/1457, could we somehow have a way for CI to measure the flakiness of new tests before they land? Something like:
- For every PR, identify tests that are added by the PR (probably tests that run in the PR’s branch that didn’t run for
main). - Run
measure-flakinesson them. - Fail CI unless the new tests pass the flakiness cutoff, on all platforms.
This obviously won’t help for existing flaky tests, but I would think that it should prevent most new flaky tests from landing on main; and it would highly motivate contributors to improve their tests, because their PRs would be blocked from landing until they did so.
It also wouldn’t help if a test becomes flaky because of changes to the API that it tests after the test has landed. But still, I think this is better than the status quo.
Related: https://github.com/nodejs/build/pull/3056 cc @nodejs/tsc