Measure test strength with mutation testing and stop encoding current behavior as specification

Abierto
  • ¿Algo no está bien?
#86 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go
Área
ci-cd, testing-qa

Línea de trabajo

Start with pkg/reportdiff/compare_test.go and TestLanguagePercentagesRetainTheirDenominatorAndLimits, then review the core package list and the Enry and go-git paths named in the proposal. Run the existing reportdiff and CLI tests to identify behavior-as-spec and silence assertions. Done means on-demand and pre-release mutation runs, tracked package results, audited tests, and diagnostic-code tables including valid-input-no-warning cases.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement follow-up

Why

82.6% statement coverage says which lines ran, not whether any assertion would notice them being wrong. Two findings from 2026-09-22 show the gap:

  • Tests that encode current behavior as the specification. TestLanguagePercentagesRetainTheirDenominatorAndLimits (pkg/reportdiff/compare_test.go) asserts Unavailable == 1 when a language appears only in the newer report. That is exactly the #10 defect (added languages reported as "unavailable"), so fixing the bug would fail the test.
  • "Silence" assertions. Many CLI tests check stderr == "" rather than asserting the specific warning codes the contract promises. A wrong warning, such as the false unsupported_gitattributes for a pattern-only line, can only be caught by an assertion that expects particular diagnostics.

Proposal

  • Mutation testing (e.g. gremlins or go-mutesting) on the core packages:
    • pkg/scanner, pkg/declarations, pkg/projects, pkg/reportdiff, pkg/environments;
    • the Enry fork's strategy chain and heuristics (third_party/go-enry/common.go, data/);
    • the go-git fork's patched read path.
    • Publish mutation scores per package, and triage surviving mutants into new assertions.
  • Spec-referenced tests. Every test that pins behavior modeled on an external system (Linguist, Git, Cargo, npm, PEP 508/440, MSBuild, SARIF) cites the governing source: gem file and line, Git docs, the Cargo reference. A test that encodes a known limitation must say so and link its issue, so that fixing the limitation is an expected test change, not a surprise.
  • Diagnostics are contract. For each module, table tests assert exact warning and diagnostic codes, both positive and negative, including "this valid input produces no warning".

Acceptance

  • Mutation testing runs on demand (workflow_dispatch or a local make target) and before releases, with results tracked over time. Nothing runs on a schedule (#92). No hard gate at first; set a floor per core package once there's a baseline.
  • An audit of existing tests for behavior-as-spec assertions, starting with the reportdiff language test above, with each one either re-specified or linked to its issue.
  • Every module has diagnostic-code table tests, including valid-input-no-warning cases (a pattern-only .gitattributes line; Cargo { workspace = true, default-features = false }).

Part of the 1.0 quality program: #83. Epic: #81.

Lenguaje dominante
Go
Estrellas
0
Forks
0
Merge medio
11 h 3 min
PR fusionados (30 d)
5

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de war-and-code/dircue

Todos los issues de war-and-code/dircue

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.