Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Add diagnostic code coverage reporting to CI

Open
  • Is something wrong?
#673 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
ci-cd, testing-qa

Research direction

The issue names no files, tests, or CI entry points. Start by locating the representative CPU and GPU CI jobs and the maintained production and integration packages they exercise. Done means readable line and practical branch coverage reports are retained, suite contributions and limitations are documented, and follow-up findings are recorded without adding a repository-wide threshold.

Written by the indexing model from the issue text.

Description

Priority-P2

Add diagnostic code coverage reporting to CI

Summary

Add code coverage reporting to the FlashDreams CI test suites so that we can understand which parts of the codebase are exercised and how that coverage is distributed.

The objective is not to require or pursue 100% coverage. Coverage should be used as a diagnostic signal to:

  • identify important code paths with little or no automated testing;
  • identify areas where multiple tests may provide substantially redundant coverage;
  • understand what is covered by CPU, GPU, application, and integration test suites; and
  • make informed decisions about where adding, consolidating, or removing tests would improve confidence and CI efficiency.

Motivation

FlashDreams has tests spread across the core package, applications, and model integrations, with different tests running in CPU and GPU CI jobs. It is currently difficult to answer basic questions such as:

  • Which production modules and critical branches are not exercised by CI?
  • Are important runtime and API paths covered only by expensive GPU tests when a focused CPU test could cover them?
  • Do multiple expensive tests exercise essentially the same code without adding meaningful behavioral confidence?
  • Which gaps are intentional because the corresponding code requires particular hardware, external services, checkpoints, or manual validation?

A repository-wide coverage baseline would make these gaps and overlaps visible. It would also help us improve the test suite based on risk and value rather than optimizing for a single aggregate percentage.

Proposed work

Generate coverage in CI

Add coverage collection to the representative CI test jobs. At minimum, include the normal CPU suite and the practical GPU suites needed to measure GPU-only runtime paths.

Coverage collection should:

  • measure FlashDreams production code, applications, and maintained integration packages;
  • report line and branch coverage where practical;
  • keep results from distinct suites or jobs identifiable;
  • combine compatible parallel and distributed test results correctly;
  • exclude tests, generated code, vendored code, and other agreed non-actionable sources; and
  • avoid materially increasing test runtime or changing test behavior.

Where collecting coverage from a job is prohibitively expensive or technically unreliable, document the limitation rather than blocking the initial rollout.

Publish useful reports

Publish a human-readable report as a CI artifact, including:

  • an aggregate repository view;
  • coverage grouped by package or major subsystem;
  • missing lines and branches; and
  • a breakdown showing which CI suite contributes coverage where feasible.

Choose whether an external coverage service is warranted or whether CI-native artifacts and summaries are sufficient. Prefer the smallest solution that provides durable reports and makes meaningful changes visible during review.

Establish a baseline and review it

Use the first complete report to identify:

  • high-risk or commonly used paths with major coverage gaps;
  • code that appears unreachable, obsolete, or difficult to test;
  • important behavior covered only incidentally by large end-to-end tests;
  • code covered only by expensive GPU jobs that could be validated more cheaply;
  • suites with significant overlap that may be candidates for consolidation; and
  • expected gaps that should be documented rather than immediately filled.

Coverage overlap alone is not proof that tests are redundant: tests may exercise the same lines while validating different behavior, inputs, hardware paths, or failure modes. Any recommendation to consolidate tests should consider assertions, scenarios, runtime cost, and defect-detection value in addition to raw coverage.

Record the findings and create focused follow-up issues for the most valuable gaps or consolidation opportunities.

Coverage policy

Do not introduce a repository-wide coverage percentage gate as part of the initial implementation. In particular, CI should not fail merely because the total is below an arbitrary target.

Once a reliable baseline exists, we can separately decide whether narrowly scoped regression safeguards are useful—for example, detecting a substantial unexpected drop in a well-understood package. Such safeguards should not encourage low-value tests or prevent deliberate deletion and simplification of code.

Out of scope

  • Reaching 100% line or branch coverage.
  • Adding tests solely to increase a headline percentage.
  • Requiring every hardware-, service-, or checkpoint-dependent path to run in ordinary CI.
  • Removing tests based only on overlapping executed lines.
  • Resolving every gap found by the initial report within this issue.

Acceptance criteria

  • Coverage is collected for the representative CPU CI suite.
  • Coverage is collected for practical GPU CI suites, or exclusions and technical limitations are documented.
  • Line coverage and, where practical, branch coverage are reported for maintained production code.
  • Reports distinguish the contributions of major CI suites or jobs where feasible.
  • Compatible results from parallel or distributed jobs can be combined into an aggregate report.
  • Tests, generated sources, vendored code, and other agreed exclusions do not distort the report.
  • A readable coverage report is retained as a CI artifact or published through an agreed reporting service.
  • The initial baseline is reviewed for major gaps, expensive incidental coverage, and potential redundant coverage.
  • Intentional or currently unavoidable gaps are documented.
  • High-value testing or consolidation opportunities are captured as focused follow-up issues.
  • No arbitrary repository-wide coverage threshold is introduced.
  • The runtime and resource overhead of coverage collection is measured and documented.
Dominant language
Python
Stars
529
Forks
69
Avg merge
2d 1h
Merged PRs (30d)
41

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/flashdreams

All issues in NVIDIA/flashdreams

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.