Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Decide the verifier's sampling: it runs at provider defaults with every generation knob unset

Aperta
  • Qualcosa non va?
#884 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
12/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java, ollama
Ambito
ai

Direzione di ricerca

Start in src/main/java/dev/thiagogonzaga/thrillhousebot/review/ai/ChatModelCustomizers.java (BLOCKING_TUNING and STREAMING_TUNING) and the sampling keys in src/main/resources/application.properties lines 108-125. The first step is a corpus measurement over the eval set from #113, comparing verdict agreement at the provider default and at temperature=0. Done means the measurement is recorded in the issue and the verifier's sampling is either changed through a separate model binding or explicitly declined.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement java spike

Background

Every model call the bot makes runs at whatever sampling the provider defaults to. ChatModelCustomizers passes temperature, top-p, the frequency and presence penalties and seed from thrillhousebot.ai.models.<model>.* to both the blocking and streaming builders, but each is applied ifPresent, and every one of those keys is commented out in application.properties (lines 120-125 are the deepseek-chat examples). #64 and #50 built that plumbing and closed; neither decided what the values should be.

For the review pass, sampling is arguably a feature: two readings of a diff can legitimately differ. For the second-pass verifier it is not. #711 measured what that costs: the audit rejected a SQL-injection candidate with correct reasoning and published the same claim as CRITICAL forty minutes later on an unchanged commit. #711's fix remembers rejections for the head, which stops that claim coming back, but the next claim is still judged by a sampled call.

The question this issue exists to answer

Should the verifier run greedy (temperature=0, and a fixed seed where the provider honours one), and what does it cost in verdict quality?

The pieces that make it non-trivial, all verified in code:

  • The verifier shares the concise named model with the final summary call and maintainer replies (ChatModelCustomizers, the concise bean). Per-model settings are per model name, not per call site, so setting temperature=0 there changes the summary's prose too. Either that is acceptable, or the verifier needs its own binding.
  • A unit test can only show the builder received the value. Whether greedy decoding makes verdicts better, worse or merely repeatable is a corpus question, which is what #113's eval/regression corpus is for.
  • seed is accepted by the OpenAI-compatible wire format but is honoured unevenly across providers, so it cannot be relied on as the only mechanism.

Change required

  • Measure first: run the verifier over the corpus at the provider default and at temperature=0, and compare verdict agreement across repeated runs of the same input as well as agreement with the graded outcome. Report both numbers in the issue before changing a default.
  • If greedy wins, set it for the verifier only, adding a separate model binding if the shared concise model makes that impossible, and document the knob in the README's per-model section.
  • If it does not, write down the measurement here and close this, so the next person asking does not have to re-run it.

Constraints

  • No change to the review pass's sampling in this issue.
  • The reasoning step-down (#839) sets reasoning_effort per call already; whatever is added must compose with it rather than duplicate it.

Acceptance criteria

  • The measurement is recorded in this issue: repeat-agreement and corpus agreement, at the default and at temperature=0.
  • A decision follows from it, applied to the verifier alone or explicitly declined.
  • If a default changes, the README per-model section and application.properties say what it is and why.

References

  • src/main/java/dev/thiagogonzaga/thrillhousebot/review/ai/ChatModelCustomizers.java (BLOCKING_TUNING, STREAMING_TUNING)
  • src/main/resources/application.properties lines 108-125
  • #711 (opposite verdicts on one commit), #113 (eval corpus), #64 and #50 (the plumbing)
Lingua principale
Java
Stelle
35
Fork
7
Merge medio
9h 23m
PR unite (30g)
154

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di devops-thiago/ThrillhouseBot

Tutte le issue di devops-thiago/ThrillhouseBot

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.