juliet_vulnerability_analyzer: the word "infer" in a comment scores 0

Aperta
  • Qualcosa non va?
#39 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, shell
Ambito
testing, tooling

Direzione di ricerca

The banned-word check lives in scoring/score.py, where any submitted .py or .sh file containing infer subtracts 100 points. Reproduce by adding # Infer external inputs from call facts. to an analyzer and running tools/judge_local.py on edgebench/tasks/juliet_vulnerability_analyzer, with and without the line. Proposed fix #35 narrows matching to command-like text; done means a comment mentioning infer no longer zeroes the score, and real Infer calls are still caught.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Task

juliet_vulnerability_analyzer

Claim

scoring/score.py subtracts 100 points if any submitted .py or .sh file contains the word infer, in any case, even in a comment. The ban targets the Infer tool, but the README itself says "Infer external inputs". This is part of Known issue 3.

Impact

Measured, six grades per judge per row, no spread:

Submission Current judge With #35 merged
Opus 5.5 analyzer whose docstring has 2. Infer the "bad" (attacker reachable) region of the program from the 0 79.59
Another analyzer, no such word 90.70 90.70
The same analyzer plus # Infer external inputs from call facts. 0 90.70
Evidence and how to reproduce

Add that comment line to any submission's analyzer.py, then grade with and without it:

python3 tools/judge_local.py --task-dir edgebench/tasks/juliet_vulnerability_analyzer --image <judge tag> --archive <submission.tar.gz> --out result.json
Fix

#35, merged, drops infer from the banned words. Scores can only rise, and only for submissions containing the word.

Limits

Other banned words in comments still zero a submission. The README's rule against calling Infer stays without a text check; Infer is not in the image and grading is offline.

Found during EdgeBench Verified, our audit of EdgeBench's tasks. Its page for this task covers a different finding.

Lingua principale
Python
Stelle
474
Fork
20
Merge medio
1g 7h
PR unite (30g)
2

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ByteDance-Seed/EdgeBench

Tutte le issue di ByteDance-Seed/EdgeBench

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.