deep-code-review: a precision/recall detection-quality eval instrument on real paired diffs (extends #61)

Aperta
  • Qualcosa non va?
#267 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
32/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
ai-infra-agents, testing-qa

Direzione di ricerca

Read #61 and the existing planted-defect probe matrix in method.md Phase 1, then inspect the references/ methods and offline run-evals.py mentioned in the issue. Define a corpus from fixing commits with paired vulnerable and clean examples, and report precision and recall under an explicit false-positive budget. Done means the accepted method, judge-bias controls for #61, and an eval-of-the-eval where a plausible but wrong review is scored as a miss are documented.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

BLUF

Perun's 184 evals are behavior / refusal / conformance checks — they do not measure the thing a review tool lives or dies on: detection quality on real diffs (recall) under a false-positive budget (precision). The field has concrete, adoptable methods for this. This issue is the measurement instrument; it extends #61 (which builds the runner) rather than duplicating it.

Concrete evidence (measurement methods, primary sources 2026-09-17)

  • Paired known-bad/known-good fixtures + FP-budget metric — PrimeVul keeps "challenging pairs" (≥80% identical vulnerable vs patched fn) and reports VD-S = false-negative-rate at false-positive-rate ≤ 0.5% — scoring detection under an explicit FP budget, not raw accuracy (arxiv 2403.18624).
  • Build the corpus from real fixing commits — CVEfixes: CVE→fixing commit→before=vulnerable, after=clean (arxiv 2107.08760). Reuses Perun's existing START_SHA / git log -S discipline.
  • Dual gate — SWE-bench's FAIL_TO_PASS (bug must be caught) + PASS_TO_PASS (clean must not regress) (arxiv 2310.06770). Perun's planted-defect probe matrix (method.md Phase 1) already is this shape — generalize it into the eval corpus.
  • LLM-as-judge controls for the live harness — position-swap, reference-guided, rubric+CoT (G-Eval, arxiv 2303.16634); MT-bench measured position/verbosity/self-enhancement bias (arxiv 2306.05685).
  • Reference-free coverage scoring — CRScore derives precision (conciseness) + recall (comprehensiveness) from static-analyzer "smells" + LLM code-claims, no human reference comments needed (arxiv 2409.19801) — and it corroborates the SAST signals of the SAST-naming issue.
  • Promote every production escape into a permanent regression fixture (Braintrust offline+online) — generalizes delivery G10's escaped-gap→regression.

Honesty note surfaced by the research: vendor "catch rates" exclude false positives by design (Greptile 82% is catch-rate only; the "11 FP/run" figure is third-party); Anthropic's "<1% marked incorrect" is undefined on its page. Precision must be measured separately — Perun should never cite a vendor catch-rate as a clean precision number.

Perun status — PARTIAL

Has: planted-defect probe matrix (a dual-gate control), principle 2 (positive-control-before-absence), good/red fixtures, offline run-evals.py, escaped-gap→regression on the delivery side (G10). Absent: a precision/recall detection instrument on real paired diffs; an FP budget; LLM-judge decorrelation/bias controls (the #61 remainder is unspecified beyond "split rubric").

Why it fits the thesis

Pure eval/measurement discipline — Perun's own domain J. Addresses its single weakest surface. No product.

Enforceable mitigation

  1. New references/ eval-harness method: paired known-bad/known-good fixtures + FP-rate-at-budget metric; corpus-from-fixing-commits construction; reference-free coverage scoring; promote-escape-to-regression.
  2. Give #61 its missing spec: LLM-as-judge controls (position-swap, reference-guided, rubric+CoT, judge≠subject decorrelation already required).
  3. Wire it to the existing planted-probe / principle-2 kernel (link, do not restate).

Acceptance

  • A documented method to build a paired known-bad/known-good corpus from a target repo's own fixing commits.
  • A precision/recall report with an explicit FP budget (not accuracy).
  • The #61 harness spec names its LLM-judge bias controls.
  • An eval-of-the-eval: a plausible-but-wrong review scores as a miss.

Distinct from

Extends #61 (the runner) — this is the metric + corpus method, not the runner. Not a duplicate of #64/#65 (eval-drift / fabrication-refusal evals), which are conformance/refusal checks.

Sources

arxiv 2403.18624 (PrimeVul) · arxiv 2107.08760 (CVEfixes) · arxiv 2310.06770 (SWE-bench) · arxiv 2303.16634 (G-Eval) · arxiv 2306.05685 (MT-bench) · arxiv 2409.19801 (CRScore) · braintrust.dev/articles/llm-evaluation-guide · greptile.com/benchmarks · claude.com/blog/code-review

Lingua principale
Python
Stelle
2
Fork
1
Merge medio
1h 28m
PR unite (30g)
572

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di protocol/perun

Tutte le issue di protocol/perun

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.