Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Decide the verifier's sampling: it runs at provider defaults with every generation knob unset

Open
  • Is something wrong?
#884 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
12/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
java, ollama
Domain
ai

Research direction

Start in src/main/java/dev/thiagogonzaga/thrillhousebot/review/ai/ChatModelCustomizers.java (BLOCKING_TUNING and STREAMING_TUNING) and the sampling keys in src/main/resources/application.properties lines 108-125. The first step is a corpus measurement over the eval set from #113, comparing verdict agreement at the provider default and at temperature=0. Done means the measurement is recorded in the issue and the verifier's sampling is either changed through a separate model binding or explicitly declined.

Written by the indexing model from the issue text.

Description

enhancement java spike

Background

Every model call the bot makes runs at whatever sampling the provider defaults to. ChatModelCustomizers passes temperature, top-p, the frequency and presence penalties and seed from thrillhousebot.ai.models.<model>.* to both the blocking and streaming builders, but each is applied ifPresent, and every one of those keys is commented out in application.properties (lines 120-125 are the deepseek-chat examples). #64 and #50 built that plumbing and closed; neither decided what the values should be.

For the review pass, sampling is arguably a feature: two readings of a diff can legitimately differ. For the second-pass verifier it is not. #711 measured what that costs: the audit rejected a SQL-injection candidate with correct reasoning and published the same claim as CRITICAL forty minutes later on an unchanged commit. #711's fix remembers rejections for the head, which stops that claim coming back, but the next claim is still judged by a sampled call.

The question this issue exists to answer

Should the verifier run greedy (temperature=0, and a fixed seed where the provider honours one), and what does it cost in verdict quality?

The pieces that make it non-trivial, all verified in code:

  • The verifier shares the concise named model with the final summary call and maintainer replies (ChatModelCustomizers, the concise bean). Per-model settings are per model name, not per call site, so setting temperature=0 there changes the summary's prose too. Either that is acceptable, or the verifier needs its own binding.
  • A unit test can only show the builder received the value. Whether greedy decoding makes verdicts better, worse or merely repeatable is a corpus question, which is what #113's eval/regression corpus is for.
  • seed is accepted by the OpenAI-compatible wire format but is honoured unevenly across providers, so it cannot be relied on as the only mechanism.

Change required

  • Measure first: run the verifier over the corpus at the provider default and at temperature=0, and compare verdict agreement across repeated runs of the same input as well as agreement with the graded outcome. Report both numbers in the issue before changing a default.
  • If greedy wins, set it for the verifier only, adding a separate model binding if the shared concise model makes that impossible, and document the knob in the README's per-model section.
  • If it does not, write down the measurement here and close this, so the next person asking does not have to re-run it.

Constraints

  • No change to the review pass's sampling in this issue.
  • The reasoning step-down (#839) sets reasoning_effort per call already; whatever is added must compose with it rather than duplicate it.

Acceptance criteria

  • The measurement is recorded in this issue: repeat-agreement and corpus agreement, at the default and at temperature=0.
  • A decision follows from it, applied to the verifier alone or explicitly declined.
  • If a default changes, the README per-model section and application.properties say what it is and why.

References

  • src/main/java/dev/thiagogonzaga/thrillhousebot/review/ai/ChatModelCustomizers.java (BLOCKING_TUNING, STREAMING_TUNING)
  • src/main/resources/application.properties lines 108-125
  • #711 (opposite verdicts on one commit), #113 (eval corpus), #64 and #50 (the plumbing)
Dominant language
Java
Stars
35
Forks
7
Avg merge
9h 23m
Merged PRs (30d)
154

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from devops-thiago/ThrillhouseBot

All issues in devops-thiago/ThrillhouseBot

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.