nodejs/build

Measure flakiness of new tests

オープン

#3,541 opened on 2023/10/25

 (4 件のコメント) (0 件のリアクション) (0 人の担当者)Jinja (180 件のフォーク)auto 404
enhancementhelp wantedquestion

Repository metrics

Stars
 (537 個のスター)
PR merge metrics
 (PR metrics pending)

説明

As discussed in https://github.com/nodejs/TSC/issues/1457, could we somehow have a way for CI to measure the flakiness of new tests before they land? Something like:

  • For every PR, identify tests that are added by the PR (probably tests that run in the PR’s branch that didn’t run for main).
  • Run measure-flakiness on them.
  • Fail CI unless the new tests pass the flakiness cutoff, on all platforms.

This obviously won’t help for existing flaky tests, but I would think that it should prevent most new flaky tests from landing on main; and it would highly motivate contributors to improve their tests, because their PRs would be blocked from landing until they did so.

It also wouldn’t help if a test becomes flaky because of changes to the API that it tests after the test has landed. But still, I think this is better than the status quo.

Related: https://github.com/nodejs/build/pull/3056 cc @nodejs/tsc

コントリビューターガイド