deep-code-review: a precision/recall detection-quality eval instrument on real paired diffs (extends #61)

Đang mở
  • Có gì không ổn?
#267 5 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
32/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Ít trao đổi
Công nghệ
python
Lĩnh vực
ai-infra-agents, testing-qa

Hướng nghiên cứu

Read #61 and the existing planted-defect probe matrix in method.md Phase 1, then inspect the references/ methods and offline run-evals.py mentioned in the issue. Define a corpus from fixing commits with paired vulnerable and clean examples, and report precision and recall under an explicit false-positive budget. Done means the accepted method, judge-bias controls for #61, and an eval-of-the-eval where a plausible but wrong review is scored as a miss are documented.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

BLUF

Perun's 184 evals are behavior / refusal / conformance checks — they do not measure the thing a review tool lives or dies on: detection quality on real diffs (recall) under a false-positive budget (precision). The field has concrete, adoptable methods for this. This issue is the measurement instrument; it extends #61 (which builds the runner) rather than duplicating it.

Concrete evidence (measurement methods, primary sources 2026-09-17)

  • Paired known-bad/known-good fixtures + FP-budget metric — PrimeVul keeps "challenging pairs" (≥80% identical vulnerable vs patched fn) and reports VD-S = false-negative-rate at false-positive-rate ≤ 0.5% — scoring detection under an explicit FP budget, not raw accuracy (arxiv 2403.18624).
  • Build the corpus from real fixing commits — CVEfixes: CVE→fixing commit→before=vulnerable, after=clean (arxiv 2107.08760). Reuses Perun's existing START_SHA / git log -S discipline.
  • Dual gate — SWE-bench's FAIL_TO_PASS (bug must be caught) + PASS_TO_PASS (clean must not regress) (arxiv 2310.06770). Perun's planted-defect probe matrix (method.md Phase 1) already is this shape — generalize it into the eval corpus.
  • LLM-as-judge controls for the live harness — position-swap, reference-guided, rubric+CoT (G-Eval, arxiv 2303.16634); MT-bench measured position/verbosity/self-enhancement bias (arxiv 2306.05685).
  • Reference-free coverage scoring — CRScore derives precision (conciseness) + recall (comprehensiveness) from static-analyzer "smells" + LLM code-claims, no human reference comments needed (arxiv 2409.19801) — and it corroborates the SAST signals of the SAST-naming issue.
  • Promote every production escape into a permanent regression fixture (Braintrust offline+online) — generalizes delivery G10's escaped-gap→regression.

Honesty note surfaced by the research: vendor "catch rates" exclude false positives by design (Greptile 82% is catch-rate only; the "11 FP/run" figure is third-party); Anthropic's "<1% marked incorrect" is undefined on its page. Precision must be measured separately — Perun should never cite a vendor catch-rate as a clean precision number.

Perun status — PARTIAL

Has: planted-defect probe matrix (a dual-gate control), principle 2 (positive-control-before-absence), good/red fixtures, offline run-evals.py, escaped-gap→regression on the delivery side (G10). Absent: a precision/recall detection instrument on real paired diffs; an FP budget; LLM-judge decorrelation/bias controls (the #61 remainder is unspecified beyond "split rubric").

Why it fits the thesis

Pure eval/measurement discipline — Perun's own domain J. Addresses its single weakest surface. No product.

Enforceable mitigation

  1. New references/ eval-harness method: paired known-bad/known-good fixtures + FP-rate-at-budget metric; corpus-from-fixing-commits construction; reference-free coverage scoring; promote-escape-to-regression.
  2. Give #61 its missing spec: LLM-as-judge controls (position-swap, reference-guided, rubric+CoT, judge≠subject decorrelation already required).
  3. Wire it to the existing planted-probe / principle-2 kernel (link, do not restate).

Acceptance

  • A documented method to build a paired known-bad/known-good corpus from a target repo's own fixing commits.
  • A precision/recall report with an explicit FP budget (not accuracy).
  • The #61 harness spec names its LLM-judge bias controls.
  • An eval-of-the-eval: a plausible-but-wrong review scores as a miss.

Distinct from

Extends #61 (the runner) — this is the metric + corpus method, not the runner. Not a duplicate of #64/#65 (eval-drift / fabrication-refusal evals), which are conformance/refusal checks.

Sources

arxiv 2403.18624 (PrimeVul) · arxiv 2107.08760 (CVEfixes) · arxiv 2310.06770 (SWE-bench) · arxiv 2303.16634 (G-Eval) · arxiv 2306.05685 (MT-bench) · arxiv 2409.19801 (CRScore) · braintrust.dev/articles/llm-evaluation-guide · greptile.com/benchmarks · claude.com/blog/code-review

Ngôn ngữ chính
Python
Star
2
Fork
1
Merge trung bình
1 giờ 27 phút
Pull request đã merge (30 ngày)
561

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của protocol/perun

Tất cả issue của protocol/perun

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.