nodejs/build

Measure flakiness of new tests

開放

#3,541 建立於 2023年10月25日

 (4 則留言) (0 個反應) (0 位負責人)Jinja (180 個分叉)auto 404
enhancementhelp wantedquestion

倉庫指標

星標
 (537 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

As discussed in https://github.com/nodejs/TSC/issues/1457, could we somehow have a way for CI to measure the flakiness of new tests before they land? Something like:

  • For every PR, identify tests that are added by the PR (probably tests that run in the PR’s branch that didn’t run for main).
  • Run measure-flakiness on them.
  • Fail CI unless the new tests pass the flakiness cutoff, on all platforms.

This obviously won’t help for existing flaky tests, but I would think that it should prevent most new flaky tests from landing on main; and it would highly motivate contributors to improve their tests, because their PRs would be blocked from landing until they did so.

It also wouldn’t help if a test becomes flaky because of changes to the API that it tests after the test has landed. But still, I think this is better than the status quo.

Related: https://github.com/nodejs/build/pull/3056 cc @nodejs/tsc

貢獻者指南