Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Make deterministic (non-LLM) audits explicitly test-only

Open
  • Is something wrong?
#505 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
cli, developer-experience

Research direction

Start by reading the darnit audit and darnit run command entry points, then trace how run headers, machine-readable output, and compliance levels are produced. Review the README and install/usage docs for the relevant descriptions. Done means deterministic runs are clearly marked as test-only, make no compliance claims or attestations, keep judgment-dependent controls pending, and the help and docs point users to the supported surfaces.

Written by the indexing model from the issue text.

Description

documentation interface-cli

Summary

The deterministic (non-LLM) audit paths -- darnit audit, darnit run, and any mode that runs controls without resolving LLM judgments -- exist to test individual handlers, passes, and controls during development. They are not a supported way to evaluate a project. The supported surfaces are the MCP server with agent skills (a coding agent answers the judgment tasks) and darnit harness (the headless driver for CI and fleets, which dispatches LLM judgments itself).

Today nothing in the output or documentation makes that distinction, so a deterministic run can look like a real compliance result.

Proposed changes

  • Label the output. Deterministic runs state in their header and in machine-readable output (run_mode: "deterministic-test") that they are a development/test run and not a compliance result. Controls that would need a judgment remain pending and are shown as such.
  • No compliance claims. Deterministic runs do not report a level as compliant and do not produce attestations, in line with conservative-by-default.
  • Help and docs. CLI help for these commands, the README, and the install/usage docs describe them as testing tools and point users to the MCP server + skills or darnit harness.
  • Focused test tooling (optional). Consider a narrower entry point for the actual use case -- running one control or one handler against a fixture repository and printing each pass's outcome and evidence -- so plugin authors have a first-class way to test hooks.

Out of scope

Removing the deterministic engine: the check engine itself is shared by every driver, and deterministic passes remain the first step of every real run.

Dominant language
Python
Stars
22
Forks
27
Avg merge
1d 20h
Merged PRs (30d)
39

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from darnitdevorg/darnit

All issues in darnitdevorg/darnit

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.