deep-code-review: a precision/recall detection-quality eval instrument on real paired diffs (extends #61)

Abierto
  • ¿Algo no está bien?
#267 5 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
32/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
python
Área
ai-infra-agents, testing-qa

Línea de trabajo

Read #61 and the existing planted-defect probe matrix in method.md Phase 1, then inspect the references/ methods and offline run-evals.py mentioned in the issue. Define a corpus from fixing commits with paired vulnerable and clean examples, and report precision and recall under an explicit false-positive budget. Done means the accepted method, judge-bias controls for #61, and an eval-of-the-eval where a plausible but wrong review is scored as a miss are documented.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

BLUF

Perun's 184 evals are behavior / refusal / conformance checks — they do not measure the thing a review tool lives or dies on: detection quality on real diffs (recall) under a false-positive budget (precision). The field has concrete, adoptable methods for this. This issue is the measurement instrument; it extends #61 (which builds the runner) rather than duplicating it.

Concrete evidence (measurement methods, primary sources 2026-09-17)

  • Paired known-bad/known-good fixtures + FP-budget metric — PrimeVul keeps "challenging pairs" (≥80% identical vulnerable vs patched fn) and reports VD-S = false-negative-rate at false-positive-rate ≤ 0.5% — scoring detection under an explicit FP budget, not raw accuracy (arxiv 2403.18624).
  • Build the corpus from real fixing commits — CVEfixes: CVE→fixing commit→before=vulnerable, after=clean (arxiv 2107.08760). Reuses Perun's existing START_SHA / git log -S discipline.
  • Dual gate — SWE-bench's FAIL_TO_PASS (bug must be caught) + PASS_TO_PASS (clean must not regress) (arxiv 2310.06770). Perun's planted-defect probe matrix (method.md Phase 1) already is this shape — generalize it into the eval corpus.
  • LLM-as-judge controls for the live harness — position-swap, reference-guided, rubric+CoT (G-Eval, arxiv 2303.16634); MT-bench measured position/verbosity/self-enhancement bias (arxiv 2306.05685).
  • Reference-free coverage scoring — CRScore derives precision (conciseness) + recall (comprehensiveness) from static-analyzer "smells" + LLM code-claims, no human reference comments needed (arxiv 2409.19801) — and it corroborates the SAST signals of the SAST-naming issue.
  • Promote every production escape into a permanent regression fixture (Braintrust offline+online) — generalizes delivery G10's escaped-gap→regression.

Honesty note surfaced by the research: vendor "catch rates" exclude false positives by design (Greptile 82% is catch-rate only; the "11 FP/run" figure is third-party); Anthropic's "<1% marked incorrect" is undefined on its page. Precision must be measured separately — Perun should never cite a vendor catch-rate as a clean precision number.

Perun status — PARTIAL

Has: planted-defect probe matrix (a dual-gate control), principle 2 (positive-control-before-absence), good/red fixtures, offline run-evals.py, escaped-gap→regression on the delivery side (G10). Absent: a precision/recall detection instrument on real paired diffs; an FP budget; LLM-judge decorrelation/bias controls (the #61 remainder is unspecified beyond "split rubric").

Why it fits the thesis

Pure eval/measurement discipline — Perun's own domain J. Addresses its single weakest surface. No product.

Enforceable mitigation

  1. New references/ eval-harness method: paired known-bad/known-good fixtures + FP-rate-at-budget metric; corpus-from-fixing-commits construction; reference-free coverage scoring; promote-escape-to-regression.
  2. Give #61 its missing spec: LLM-as-judge controls (position-swap, reference-guided, rubric+CoT, judge≠subject decorrelation already required).
  3. Wire it to the existing planted-probe / principle-2 kernel (link, do not restate).

Acceptance

  • A documented method to build a paired known-bad/known-good corpus from a target repo's own fixing commits.
  • A precision/recall report with an explicit FP budget (not accuracy).
  • The #61 harness spec names its LLM-judge bias controls.
  • An eval-of-the-eval: a plausible-but-wrong review scores as a miss.

Distinct from

Extends #61 (the runner) — this is the metric + corpus method, not the runner. Not a duplicate of #64/#65 (eval-drift / fabrication-refusal evals), which are conformance/refusal checks.

Sources

arxiv 2403.18624 (PrimeVul) · arxiv 2107.08760 (CVEfixes) · arxiv 2310.06770 (SWE-bench) · arxiv 2303.16634 (G-Eval) · arxiv 2306.05685 (MT-bench) · arxiv 2409.19801 (CRScore) · braintrust.dev/articles/llm-evaluation-guide · greptile.com/benchmarks · claude.com/blog/code-review

Lenguaje dominante
Python
Estrellas
2
Forks
1
Merge medio
1 h 27 min
PR fusionados (30 d)
561

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de protocol/perun

Todos los issues de protocol/perun

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.