ncu-ci walk: exclude PR-modified tests from the flaky count

Open
#1,164 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
65/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
javascript
Domain
ci-cd, testing

Research direction

Start with the ncu-ci walk logic that calculates flaky counts and reliability reports, then trace how each occurrence is associated with the tested PR revision. Use the test-ffi-fast-buffer.js example and CI build 75799 as the reference case; done means occurrences whose test file was modified by the PR no longer contribute to the flaky count.

Written by the indexing model from the issue text.

Description

Current Behavior

Failures currently contribute to the flaky count even when the PR modified the failing test. Such failures are likely caused by the PR and should not count as independent flaky occurrences.

For example, CI build 75799 failed test-ffi-fast-buffer, but the PR modified test/ffi/test-ffi-fast-buffer.js. Excluding this occurrence would leave only one qualifying PR, so the test would not appear in the reliability report:

https://github.com/nodejs/reliability/blob/main/reports/2026-08-15.md

Expected Behavior

Exclude an occurrence from the flaky count when its test file was modified by the tested PR revision.

Dominant language
JavaScript
Stars
313
Forks
137
Avg merge
2d 8h
Merged PRs (30d)
24

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nodejs/node-core-utils

All issues in nodejs/node-core-utils

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.