juliet_vulnerability_analyzer: the word "infer" in a comment scores 0
维护者通常 2 天内回复
还没有人认领这个 Issue。
评估
调研方向
The banned-word check lives in scoring/score.py, where any submitted .py or .sh file containing infer subtracts 100 points. Reproduce by adding # Infer external inputs from call facts. to an analyzer and running tools/judge_local.py on edgebench/tasks/juliet_vulnerability_analyzer, with and without the line. Proposed fix #35 narrows matching to command-like text; done means a comment mentioning infer no longer zeroes the score, and real Infer calls are still caught.
由索引模型根据 Issue 内容生成。
描述
Task
juliet_vulnerability_analyzer
Claim
scoring/score.py subtracts 100 points if any submitted .py or .sh file contains the word infer, in any case, even in a comment. The ban targets the Infer tool, but the README itself says "Infer external inputs". This is part of Known issue 3.
Impact
Measured, six grades per judge per row, no spread:
| Submission | Current judge | With #35 merged |
|---|---|---|
Opus 5.5 analyzer whose docstring has 2. Infer the "bad" (attacker reachable) region of the program from the |
0 | 79.59 |
| Another analyzer, no such word | 90.70 | 90.70 |
The same analyzer plus # Infer external inputs from call facts. |
0 | 90.70 |
Evidence and how to reproduce
Add that comment line to any submission's analyzer.py, then grade with and without it:
python3 tools/judge_local.py --task-dir edgebench/tasks/juliet_vulnerability_analyzer --image <judge tag> --archive <submission.tar.gz> --out result.json
Fix
#35, merged, drops infer from the banned words. Scores can only rise, and only for submissions containing the word.
Limits
Other banned words in comments still zero a submission. The README's rule against calling Infer stays without a text check; Infer is not in the image and grading is offline.
Found during EdgeBench Verified, our audit of EdgeBench's tasks. Its page for this task covers a different finding.
- 主要语言
- Python
- 星标
- 474
- 派生
- 20
- 平均合并
- 3 天 6 小时
- 30 天内合并 PR
- 3
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
ByteDance-Seed/EdgeBench 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 73/100
ByteDance-Seed/EdgeBench#48 ·
维护者通常 2 天内回复
-
FFmpeg: public result hides whether failure occurred before correctness checks可能已有人在做 @loopx-agent 于 2 天前认领。 未关闭
难度 4/5 3-5 天 新手友好度 35/100
ByteDance-Seed/EdgeBench#57 ·
维护者通常 2 天内回复
-
难度 4/5 3-5 天 新手友好度 54/100
ByteDance-Seed/EdgeBench#52 ·
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 35/100
ByteDance-Seed/EdgeBench#51 ·
维护者通常 2 天内回复
-
难度 4/5 3-5 天 新手友好度 48/100
ByteDance-Seed/EdgeBench#50 ·
维护者通常 2 天内回复
查看 ByteDance-Seed/EdgeBench 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 83/100
-
难度 2/5 1-3 小时 新手友好度 87/100
维护者通常 7 天内回复
-
good first issue
难度 2/5 1-3 小时 新手友好度 78/100
MTSWebServices/RecTools#330 ·
-
area: web priority: P1 story
难度 2/5 1-3 小时 新手友好度 76/100
nfiasco/Quant-Time-Machine#151 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
autowarefoundation/autoware_carla_scenario#2019 ·
维护者通常 1 天内回复