juliet_vulnerability_analyzer: the word "infer" in a comment scores 0

Abierto
  • ¿Algo no está bien?
#39 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, shell
Área
testing, tooling

Línea de trabajo

The banned-word check lives in scoring/score.py, where any submitted .py or .sh file containing infer subtracts 100 points. Reproduce by adding # Infer external inputs from call facts. to an analyzer and running tools/judge_local.py on edgebench/tasks/juliet_vulnerability_analyzer, with and without the line. Proposed fix #35 narrows matching to command-like text; done means a comment mentioning infer no longer zeroes the score, and real Infer calls are still caught.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Task

juliet_vulnerability_analyzer

Claim

scoring/score.py subtracts 100 points if any submitted .py or .sh file contains the word infer, in any case, even in a comment. The ban targets the Infer tool, but the README itself says "Infer external inputs". This is part of Known issue 3.

Impact

Measured, six grades per judge per row, no spread:

Submission Current judge With #35 merged
Opus 5.5 analyzer whose docstring has 2. Infer the "bad" (attacker reachable) region of the program from the 0 79.59
Another analyzer, no such word 90.70 90.70
The same analyzer plus # Infer external inputs from call facts. 0 90.70
Evidence and how to reproduce

Add that comment line to any submission's analyzer.py, then grade with and without it:

python3 tools/judge_local.py --task-dir edgebench/tasks/juliet_vulnerability_analyzer --image <judge tag> --archive <submission.tar.gz> --out result.json
Fix

#35, merged, drops infer from the banned words. Scores can only rise, and only for submissions containing the word.

Limits

Other banned words in comments still zero a submission. The README's rule against calling Infer stays without a text check; Infer is not in the image and grading is offline.

Found during EdgeBench Verified, our audit of EdgeBench's tasks. Its page for this task covers a different finding.

Lenguaje dominante
Python
Estrellas
474
Forks
20
Merge medio
1 d 7 h
PR fusionados (30 d)
2

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ByteDance-Seed/EdgeBench

Todos los issues de ByteDance-Seed/EdgeBench

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.