nodejs/build

Measure flakiness of new tests

开放

#3,541 创建于 2023年10月25日

 (4 条评论) (0 个反应) (0 位负责人)Jinja (180 个派生)auto 404
enhancementhelp wantedquestion

仓库指标

星标
 (537 个星标)
PR 合并指标
 (PR 指标待抓取)

描述

As discussed in https://github.com/nodejs/TSC/issues/1457, could we somehow have a way for CI to measure the flakiness of new tests before they land? Something like:

  • For every PR, identify tests that are added by the PR (probably tests that run in the PR’s branch that didn’t run for main).
  • Run measure-flakiness on them.
  • Fail CI unless the new tests pass the flakiness cutoff, on all platforms.

This obviously won’t help for existing flaky tests, but I would think that it should prevent most new flaky tests from landing on main; and it would highly motivate contributors to improve their tests, because their PRs would be blocked from landing until they did so.

It also wouldn’t help if a test becomes flaky because of changes to the API that it tests after the test has landed. But still, I think this is better than the status quo.

Related: https://github.com/nodejs/build/pull/3056 cc @nodejs/tsc

贡献者指南