`eval_clean_hashes` is missing or stale for some `task_cleanroom_v6` images
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 58/100
调研方向
从 src/programbench/data/tasks/*/task.yaml、eval_batch.py 和 Evaluator._remove_hashed_files() 开始。对受影响的 task_cleanroom_v6 镜像运行提供的 docker sha256sum 命令，并将每个 digest 与其元数据进行比较。当每个真实任务都有当前且格式正确的去重哈希，并且针对重新构建镜像的更新流程已记录或自动化时，即表示完成。
由索引模型根据 Issue 内容生成。
描述
Summary
eval_clean_hashes is intended to remove byte-for-byte copies of the gold executable from a submission before compile.sh runs. However, some task metadata does not define the field, and some existing values do not match the reference executable currently shipped in the corresponding task_cleanroom_v6 image.
This makes the metadata incomplete for downstream evaluators that rely on eval_clean_hashes to scrub renamed copies of the reference executable.
Current state
At repository commit 963063c:
- 201 task directories total (including
testorg__calculator.abc1234) - 40 task YAML files do not define
eval_clean_hashes - 161 define at least one hash
- none define an explicitly empty list
Excluding the test fixture, 39 of the 200 real tasks are missing the field.
In addition, a non-empty list does not necessarily contain the SHA-256 of the executable in the current task_cleanroom_v6 image.
Examples observed:
| Task | Hash in task.yaml |
Current task_cleanroom_v6 reference SHA-256 |
|---|---|---|
facebookresearch__fasttext.1142dc4 |
missing | 80310c0ae4d92165b7a94b17439a1931d99567ae4c1d54706bb557e87bdcd51c |
halitechallenge__halite.822cfb6 |
missing | 9d5e3ec763bbfa7c1091054c3a746ee25af7d4368edc4395eb3e3d8f7f9feac3 |
tomnomnom__gron.88a6234 |
missing | 0270943958c042821654a1e19608848a8cf14cd1136fde4726e8a1821cdf4cc7 |
rs__curlie.5dfcbb1 |
e7e4f84ba194306ef782853d8b77788dd4674a6baad19c878885bb08c2db0d35 |
7ebf850e7d8d0def143b72a1d6d0ebf4e54d8212c515c2317f06e4e8f3f642f0 |
wfxr__csview.8ac4de0 |
e07420f640d1e6a0c047cab49431d1b561f661241efe111a053ff5dde1119d09 |
758ef03ba091d6a7c77d32e60aaf484aca23532aba3eea6a51bdf25232c5f5ce |
As a control, xorg62__tty-clock.f2f847c does contain the current reference hash (cd400708...).
Reproduction
For a task image:
docker run --rm \
--network none \
--user root \
--entrypoint sha256sum \
programbench/rs_1776_curlie.5dfcbb1:task_cleanroom_v6 \
/workspace/executable
Compare the resulting digest with:
src/programbench/data/tasks/rs__curlie.5dfcbb1/task.yaml
The evaluator consumes this metadata in eval_batch.py and recursively removes matching files in Evaluator._remove_hashed_files() before compilation. The evaluator comment describes these hashes as protection against byte-for-byte copies of the gold binary, so including the current cleanroom reference hash appears consistent with the existing field semantics.
Suggested fix
- Compute the SHA-256 of
/workspace/executablefor every currenttask_cleanroom_v6image. - Append it to that task's
eval_clean_hashes, preserving historical hashes and deduplicating the list. - Add a maintenance command or release script that updates these values whenever cleanroom images are rebuilt.
- Validate that every real task has at least one well-formed 64-character SHA-256 value.
- Ideally pin cleanroom images by digest, or verify/update the hashes as part of the image publication workflow, so mutable tags cannot silently drift from task metadata.
I can prepare a follow-up PR to populate the current hashes if this approach matches the intended use of eval_clean_hashes.
- 主要语言
- Python
- 星标
- 928
- 派生
- 67
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
facebookresearch/ProgramBench 的其他 Issue
-
难度 3/5 1-2 天 新手友好度 68/100
-
难度 4/5 3-5 天 新手友好度 55/100
-
难度 4/5 3-5 天 新手友好度 35/100
-
难度 4/5 3-5 天 新手友好度 48/100
-
难度 5/5 一周以上 新手友好度 25/100
facebookresearch/ProgramBench#50 · 1 条评论 ·
查看 facebookresearch/ProgramBench 的全部 Issue
相似的 Issue
-
customer-reported
难度 2/5 1-3 小时 新手友好度 68/100
Azure/azure-cli#34150 · 1 条评论 ·
维护者通常 1 天内回复
-
community-request
难度 1/5 1 小时以内 新手友好度 95/100
NVIDIA-NeMo/Curator#2464 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 88/100
WeblateOrg/translation-finder#1099 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
trezor/trezor-firmware#7997 ·
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 88/100
维护者通常 1 天内回复