Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

feat(review): execution oracle — compile and run the diff's touched tests to ground deterministic claims

Open
  • Is something wrong?
#96 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
java
Domain
ai, testing-qa

Research direction

Start by reading the review flow under src/main/java/.../review/** and trace how findings and verdicts are produced; the issue says there is no execution there today. Work out how touched tests could be compiled and run in an isolated sandbox with time and resource limits, then connect results to claims that depend on execution. Done means deterministic claims are grounded in test execution, with sandbox safety and scope defined.

Written by the indexing model from the issue text.

Description

enhancement java
Problem Statement

The bot reasons about the diff purely statically — there is no execution anywhere in the review path (no ProcessBuilder, no mvn, no sandbox). Its defenses are mature but all aimed at one failure mode — hallucination: the generator self-checks, the skeptical FindingVerifier, the deterministic FindingQuoteValidator (whose javadoc dogfoods "broken type inference" / "invalid syntax" phantom quotes), and dedup. They answer "did the model make something up?"

The PR #84 misses are the opposite failure — miscalculation, not fabrication:

  1. An off-by-one: the model counted split("\n",-1) lines by hand and pinned "expects 5, should be 4" to the wrong test (ReviewDiffFormatterTest.java:194). The quote validator passed it (the text is present), and the FindingVerifier — being the same modality — re-did the same arithmetic and agreed.
  2. A green verdict it could not actually verify (see #A).

LLMs are weakest exactly here: counting, arithmetic, and "will this compile / pass." No prompt tuning fully closes that — it needs a deterministic oracle in the loop.

Proposed Solution

For deterministically-checkable claims ("this assertion is wrong", "this won't compile", "this test fails"), run the touched test files in an ephemeral sandbox and use the real result as ground truth. Even a narrow version — compile the changed sources and run only the tests in the diff — would catch both #84-class misses. Gate any finding/verdict that rests on such a claim on the execution signal.

Alternatives Considered
  • Static-only (current): smallest footprint, but no ground truth for mechanical claims.
  • #34 (linters / static analysis / SARIF) and #59 (failing-CI context in the prompt): complementary deterministic signals, but neither actually executes the changed tests, which is the only thing that grounds "expects N, should be M" / "this test fails."
  • A full build of the whole repo per review — heavier and slower than compiling + running just the diff's tests.
Priority

Important

Additional Context
  • No execution today anywhere under src/main/java/.../review/**.
  • Relates to #34 and #59; the principled long-term answer to the #84 dogfood misses. Bigger lift: sandbox isolation, time/resource caps, and untrusted-code security need design.
Code of Conduct
  • I have searched for existing issues and this is not a duplicate.
Dominant language
Java
Stars
35
Forks
7
Avg merge
9h 23m
Merged PRs (30d)
154

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from devops-thiago/ThrillhouseBot

All issues in devops-thiago/ThrillhouseBot

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.