Nix: treefmt's `nix fmt -- --ci` is a formatting gate (and treefmt.toml says what runs: statix, deadnix, nixfmt); a `git diff --quiet` in an `if` condition is not a failing step; devenv git-hooks and `devenv shell -- cmd`

The false positive is in project_validator/facts/formatting.py around line 65, where git diff --quiet in an if condition counts as a failing step; the Nix checks are in project_validator/ecosystems/nix.py, and project_validator/tools/pre_commit.py covers prek. Start by reproducing the four workflow and flake snippets in the issue as corpus cases. Done when nix fmt -- --ci passes, the if git diff case no longer earns PASS, treefmt.toml tools are recognised, and devenv shell -- prek is credited.

agenix: FAIL "CI runs nix fmt but only to rewrite files" although nix fmt -- --ci (treefmt: --fail-on-change --no-cache) fails on change; home-manager's PASS is credited to a bot job whose git diff --quiet is an if condition. home-manager's and disko's treefmt.toml name statix, deadnix, nixf-diagnose, nixfmt and shellcheck, all run by nix fmt, yet static-analysis FAILs "no Nix lints found". home-manager's nmt test suite is exposed as flake packages, not checks ("no tests found" is wrong even though CI does not run it). cachix/devenv defines its hooks as devenv git-hooks and runs devenv shell -- prek run -a / devenv shell -- cargo nextest run: devenv shell -- is a launcher and prek is known only with a .pre-commit-config.yaml.

Evidence from the corpus audit

ryantm/agenix, nix-community/home-manager / code-formatting

• kind: FALSE_NEGATIVE
• actual: agenix FAIL: Nix: CI runs nix fmt but only to rewrite files; nothing fails when they are not formatted; home-manager's real check (nix fmt -- --ci, test.yml) recorded as format: [apply] in --facts
• expected: PASS: treefmt's --ci flag enables --fail-on-change (and --no-cache); both projects' formatter output is treefmt
• evidence: ryantm__agenix/.github/workflows/ci.yaml:20: - run: nix fmt . -- --ci; nix-community__home-manager/.github/workflows/test.yml:80: run: nix fmt -- --ci; nix-community__home-manager/home-manager/formatter.nix:14: pkgs.treefmt.withConfig {; project_validator/ecosystems/nix.py:84: checks=("--check", "-c", "--fail-on-change") (no --ci)
• repro: flake.nix with formatter.x86_64-linux + workflow step run: nix fmt -- --ci → FAIL "only to rewrite files" (reproduced in audit/scratch/fx_a, fx_b)
• confidence: high

nix-community/home-manager / code-formatting

• kind: FALSE_POSITIVE
• actual: PASS component Nix: CI enforces the nix fmt style, credited (per --facts, ToolRun origin .github/workflows/update-maintainers.yml:jobs.update-maintainers.steps[7].run) to a scheduled bot job that formats one generated file and commits it
• expected: the PASS should come from test.yml's nix fmt -- --ci (see previous finding); the update-maintainers run formats only all-maintainers.nix and its git diff --quiet is an if condition, not a failing step
• evidence: .github/workflows/update-maintainers.yml:80: nix fmt all-maintainers.nix; :87: if git diff --quiet all-maintainers.nix; then; project_validator/facts/formatting.py:65: if argv[1] == "commit" or (argv[1] == "diff" and has_option(argv, "--exit-code", "--quiet")): (a git diff --quiet used as an if condition counts as "fails when changed")
• repro: workflow with steps run: nix fmt one-file.nix then run: |\n if git diff --quiet one-file.nix; then echo same; else git commit -am x; fi → PASS "CI enforces the nix fmt style"
• confidence: high

nix-community/home-manager, nix-community/disko / static-analysis, compiler-warnings

• kind: UNSUPPORTED
• actual: home-manager FAIL: Nix: no static analyzer runs and Nix: no Nix lints found; enable at least statix, deadnix
• expected: the repo's treefmt.toml (what nix fmt runs) names statix, deadnix and nixf-diagnose, and nix fmt -- --ci runs them failing on change; treefmt.toml is a plain file the validator could read to learn what nix fmt runs (disko has the same layout with nixfmt, deadnix and shellcheck, but its CI never runs them)
• evidence: nix-community__home-manager/treefmt.toml:8-9: [formatter.statix]\ncommand = "treefmt-statix"; :13-15: [formatter.deadnix]\ncommand = "deadnix"\noptions = [ "--edit" ]; home-manager/formatter.nix:21: settings = pkgs.lib.importTOML ../treefmt.toml;; nix-community__disko/treefmt.toml:5-11 (deadnix, shellcheck); no treefmt reader in project_validator/ (grep -rn treefmt hits only a docstring in ecosystems/nix.py:83)
• repro: treefmt.toml with [formatter.deadnix] command = "deadnix" + workflow run: nix fmt -- --ci → static-analysis "no static analyzer runs"
• confidence: medium (statix runs as statix fix and deadnix with --edit, so under --ci they fail by changing files, which is enforcement but not quite "a check run")

nix-community/home-manager / tests-run

• kind: UNSUPPORTED
• actual: FAIL component: Nix: no tests found (flake checks (a checks output in flake.nix))
• expected: tests exist: home-manager's nmt-based suite in tests/ (hundreds of module tests), exposed through the flake as packages/chunks rather than checks (CI in the repo does not run them, so FAIL stands; "no tests found" is wrong)
• evidence: tests/default.nix:10-13: nmtSrc = fetchTarball {\n url = "https://git.sr.ht/~rycee/nmt/archive/v0.5.1.tar.gz";; flake.nix:84-88: tests = import ./tests {\n inherit pkgs;\n enableBig = true;\n };\n allTests = lib.attrNames tests.build;
• repro: unclear (nmt is specific to home-manager)
• confidence: low

cachix/devenv / code-formatting, static-analysis, tests-run (Rust)

• kind: UNSUPPORTED
• actual: FAIL: Rust: CI never runs rustfmt; Rust: no static analyzer runs; ci-build-types Rust: no OS/Rust toolchain combination found in CI configuration
• expected: formatting and lints are defined as devenv git-hooks (nixfmt, nixpkgs-fmt, rustfmt, ...) run by devenv shell -- prek run -a in CI and committed back; tests run as devenv shell -- cargo nextest run; the validator knows prek only with a .pre-commit-config.yaml, and does not follow devenv shell -- <command> / devenv test
• evidence: devenv.nix:158-178: git-hooks =\n ... package = pkgs.prek;\n ... rustfmt.enable = true;; .github/workflows/auto-format.yml:28: run: devenv shell -- prek run -a || true (then EndBug/add-and-commit@v11); .github/workflows/build.yml:132: ${{ steps.build-devenv.outputs.path }}/bin/devenv --system ${{ inputs.system }} shell -- cargo nextest run -j 4 --features devenv/test-all --retries 3; project_validator/tools/pre_commit.py:1: pre-commit (and prek, which reads the same configuration)
• repro: devenv.nix with git-hooks.hooks.rustfmt.enable = true; + workflow run: devenv shell -- prek run -a → rustfmt not credited
• confidence: medium (the ${{ steps...path }}/bin/devenv program is legitimately variable; devenv shell -- ... in auto-format.yml and generate.yml is literal)

Found by the regression-corpus audit (235 real repositories, one or more per ecosystem profile; corpus and expected results in corpus/). Evidence is quoted from the audited checkouts at the commits pinned in corpus/projects.toml; <checkouts>/<owner>__<repo> is such a checkout.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ByTXNqDWdM7HzezrFtEvQt