ci: evaluate and adopt mutation testing (gremlins, go-mutesting, ooze)

Start by reviewing the measured Gremlins baseline and the requirements and verification checklist in this issue; inspect the CI workflow location from related issue #96 and the existing base and b2 tests. Re-run the baseline, compare the Avito go-mutesting results, then document the tool choice and add the configuration, local command, and advisory PR and weekly workflows. Done means the listed verification checks pass, reports are visible as summaries and artifacts, and surviving mutants are triaged.

Evaluate and adopt mutation testing for blazer, so we can tell whether the unit tests actually pin behavior instead of only executing it. This issue records the candidate tools with pros and cons, the constraints specific to this repo, a baseline measurement, the requirements, the tasks, and how a reviewer verifies that the implementing PR delivers what is needed.

Why

Line coverage says code ran. Mutation testing changes the code (flip a condition, drop an assignment, change an operator) and checks that some test fails. A mutant that survives means no test would notice that bug.

This already bit us. While reviewing #47 (preserve CORS and file lock on bucket create), I broke the real change in base/base.go five different ways by hand (drop corsRules or fileLockEnabled from the request, ignore fileLockConfiguration on create, ignore it on list, drop CORSRules on list read). Every test stayed green, because the PR's tests only exercised the b2 fakes and never the base request and response mapping. Adding a base level httptest suite fixed it, and all five mutations were then caught. That was a manual loop. A tool does the same check across the whole repo and on every PR.

Constraints specific to blazer

• Zero third-party dependencies. go.sum is empty and go.mod says go 1.18. A tool must be a CLI installed in CI, not a require in go.mod. This rules out anything imported as a library unless it lives in a separate module (blazer already uses go.work for bin/b2keys).
• Most behavior is only covered by live tests. The integration tests need B2_ACCOUNT_ID and B2_ACCOUNT_KEY and skip without them, so mutation runs in CI will see mostly uncovered code until more hermetic tests exist. "Not covered" will dominate the first reports, and that is useful information, not a tool failure.
• Flaky tests. go test -race fails intermittently on TestReuploadPart* (known race, also on master). Mutation tools treat any failing test as a kill, so a flaky test produces false kills. Run mutation without -race and re-check the baseline for stability.
• Run time. Every mutant runs a go test. Full runs must be scheduled, and PR runs must be limited to changed code.
• Tool Go versions. The tools below declare go 1.25 or newer, so they need a current toolchain in CI. That is fine for a CLI, but not for a library dependency in a go 1.18 module.
• Related work: #96 (CI migration, where the workflow lives), #93 (zizmor, which will lint the workflow), #92 (rename master to main).

Candidates

Facts below were checked on 2026-10-06 against each repository and the Go module proxy.

Tool	Version, activity	How it runs	Pros	Cons
Gremlins (go-gremlins/gremlins)	v0.6.0 (2025-12-06), 439 stars, Apache-2.0, last push 2026-06-26, needs Go 1.25 to build	CLI: gremlins unleash	Coverage guided: only mutates code that tests cover, so it is faster and less noisy. --diff <ref> tests only mutants inside changed code, ideal for PRs. Efficacy and mutant coverage thresholds with non-zero exit. .gremlins.yaml, --workers, --timeout-coefficient, --exclude-files, --dry-run, --tags. 11 mutation types (5 on by default, 6 opt-in). Official GitHub Action go-gremlins/gremlins-action. Installs as a binary, so go.mod is untouched.	Pre-1.0: the README warns that flags and config can change between minor releases. Needs tuning: my first untuned run on base reported 31 of 32 mutants as timed out, and only --timeout-coefficient 20 produced real results. Reports nothing about code the tests do not reach (shown as "not covered"). No per-line suppression found in the docs reviewed.
Ooze (gtramontina/ooze)	v0.2.0, 290 stars, MIT, last push 2026-09-17, needs Go 1.25	Library: a Go test file behind a mutation build tag, configured in Go (WithMinimumThreshold, WithTestCommand, Parallel)	Runs through plain go test, no separate CLI to install. Prints a diff of each living mutant. Configurable test command and minimum score.	Imported as a dependency, so it adds a require to go.mod (breaks the zero-dependency property) and cannot live in a go 1.18 module; it would need its own module in go.work. Runs the whole test command for every mutant. Default threshold is 1.0, and Parallel defaults to false. Only mutates files in the current platform build context. No GitHub releases, only proxy tags.
go-mutesting, Avito fork (avito-tech/go-mutesting)	v2.3.1 (2025-12-26), 277 stars, MIT, last push 2026-10-02, go.mod declares go 1.27	CLI: go-mutesting ./...	Most mutators of the three (arithmetic, branch, loop, expression and statement removal). --exec runs your own test script. Blacklist file of known false positives. Comment annotations to exclude code (// mutator-disable-next-line, // mutator-disable-func). Actively maintained.	Its README says mutation testing "can generate many false positives" and tells you to curate a blacklist. Needs the newest toolchain. Not coverage guided as far as I could tell, so expect more mutants and longer runs. Text output only.
go-mutesting, original (zimmski/go-mutesting)	v1.2 (2021-06-10), last push 2024-07-04	CLI	Historical reference.	Effectively unmaintained. The Avito fork supersedes it. Do not use.
Gooze (gooze-dev/gooze)	v1.0.1 (2026-06-21), 3 stars	CLI	New.	Too young and too small to depend on. Not recommended.
Manual mutation checks (status quo)	n/a	By hand during review	No tooling, targeted.	Does not scale, depends on the reviewer remembering to do it. This is what found the gap in #47.

Provisional recommendation, to be confirmed by the evaluation tasks below: Gremlins as the primary tool (CLI, coverage guided, diff mode for PRs, thresholds, GitHub Action), run as an advisory job, with go-mutesting (Avito) evaluated once as a cross-check on base and b2. Do not adopt Ooze unless a separate module is acceptable. Do not use the original go-mutesting or Gooze.

Baseline (measured, not guessed)

Gremlins v0.6.0 on master at 8257542, Go 1.26.5, darwin/arm64, unit tests only (no credentials), with --timeout-coefficient 20 --workers 2:

Package	Killed	Lived	Not covered	Timed out	Test efficacy	Mutator coverage	Duration
base	23	9	126	0	71.88%	20.25%	about 15 seconds
b2	98	55	166	14	64.05%	47.96%	4 minutes 57 seconds

The 55 surviving mutants in b2 and 9 in base are the first triage list. The b2 run time (almost 5 minutes for one package) is why PR runs must use --diff and the full run must be scheduled.

Mutator coverage is the share of mutants that tests reach at all. Efficacy is the share of reached mutants that tests kill. Both numbers are per package, and the low coverage is expected because the integration tests skip without B2 credentials.

Requirements

1. Mutation testing runs in CI for base and b2 using a CLI tool, with no change to go.mod or go.sum.
2. The first rollout is advisory (does not block merge). Blocking thresholds are a separate, later decision.
3. PR runs are limited to changed code (Gremlins --diff), and a full run is scheduled weekly.
4. The run is hermetic: no B2 credentials, no network, no -race, and a stable test baseline (flaky tests are fixed or excluded before measuring).
5. Results are visible without reading logs: a job summary with killed, lived and not covered counts, plus the list of surviving mutants as a downloadable artifact.
6. Generated and non-library code is excluded (internal/b2assets, examples, bin, test files).
7. The choice of tool, the comparison, and how to run it locally are documented in the repo.
8. The workflow follows the repo hardening rules: least-privilege permissions, third-party actions pinned to commit SHAs, and the Dependabot guard if: ${{ github.actor != 'dependabot[bot]' && github.event.pull_request.user.login != 'dependabot[bot]' }} on every job in a workflow that has a pull_request trigger.

Tasks

• Re-run the baseline on base and b2 with Gremlins and record the numbers in this issue (the b2 row above may need refreshing after #47, #74 and #79 merge).
• Run go-mutesting (Avito fork) once on base and b2, and compare runtime, number of mutants, noise (false positives) and usability against Gremlins. Record the result in the comparison document.
• Confirm Ooze's constraints (separate module, go 1.25) and record the decision to reject or defer it.
• Decide the tool and record it with the evidence (an ADR or a section in CONTRIBUTING.md).
• Add .gremlins.yaml with the enabled mutation types, exclude-files, timeout-coefficient, and thresholds set at or just below the measured baseline (so the job is green on day one and never regresses).
• Add a make or script target (for example scripts/mutation.sh) so a contributor can run the same command locally.
• Add the workflow (PR diff mode plus weekly full run) with a job summary and an artifact of lived mutants. It is advisory (continue-on-error: true or a non-required check).
• Triage the surviving mutants in base and b2: for each, either add a test that kills it, mark it as an accepted false positive with a reason, or open an issue. Prioritize the request and response mapping in base/base.go.
• Document how to read a report: what "lived", "not covered" and "timed out" mean and what to do about each.
• Decide, after four weeks of data, whether to make the check blocking and at which threshold.

How to verify the implementing PR

A reviewer can check each requirement with these commands and artifacts.

1. No new dependency: git diff origin/master -- go.mod go.sum is empty.
2. Config exists and is sane: .gremlins.yaml is present, gremlins unleash --dry-run ./base ./b2 runs and lists mutants, and the configured thresholds are not above the recorded baseline.
3. Reproducible locally: following the documented command on a clean checkout (with Go 1.25 or newer installed) produces the same killed, lived and not covered counts as the PR description, within a small tolerance.
4. Hermetic: the workflow sets no B2 secrets, does not pass -race, and the run succeeds with B2_ACCOUNT_ID and B2_ACCOUNT_KEY unset.
5. Detects a real regression (canary): on a scratch branch, remove CORSRules: corsRules from the CreateBucketRequest literal in base.CreateBucket and run the tool. It must report that mutant as lived on master before #47's base tests, and as killed after them. The PR description must show this result, or include a script that reproduces it. This proves the tool would have caught the gap that motivated this issue.
6. PR diff mode works: a PR that changes one function only mutates mutants inside that function (check the mutant count in the job summary against the size of the diff).
7. Scheduled run exists: the workflow has a weekly schedule trigger and a workflow_dispatch trigger.
8. Outputs are visible: the job summary shows killed, lived and not covered counts, and an artifact contains the lived mutants with file and line.
9. Advisory, not blocking: the job is not listed as a required check in branch protection.
10. Workflow hardening: permissions is read-only, every third-party action is pinned to a full commit SHA, and every job in a workflow with a pull_request trigger has the Dependabot guard. If #93 has landed, zizmor passes on the new workflow.
11. Documentation: the comparison (this table or a link to it), the chosen tool and why, the local command, how to read a report, and how to suppress or accept a surviving mutant are all documented, with no em dashes.
12. Triage is recorded: each surviving mutant in base and b2 is either killed by a new test, accepted with a written reason, or linked to an issue.

Out of scope

• Making mutation testing a required, blocking check (decided later from real data).
• Raising unit test coverage across the repo (this issue only tells us where the gaps are; closing them is separate work).
• Adding a mutation library as a go.mod dependency.

Related: #96 (GitHub Actions CI), #93 (zizmor), #92 (rename master to main), #47 (the PR whose review exposed the gap).