csview: evaluator compatibility issues resolved — unchanged submission reaches official SOLVED (✅ 335 tests)
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 35/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 冷清
- 技术栈
- python
- 领域
- testing-qa
调研方向
首先检查围绕 branch run.sh、pytest/libtmux 兼容性、tests.json 以及与 eval.tests.* 的 testcase 匹配所描述的 evaluator 行为。运行指定的 41-test evaluator 回归测试套件,并检查最终的 eval.json;完成的标准是复现兼容性发现,并在不更改 submission 的情况下独立验证所报告的 335-test 结果。
由索引模型根据 Issue 内容生成。
描述
Update: we have now completed an end-to-end reproduction and isolated the evaluator-side causes affecting wfxr__csview.8ac4de0.
Using the same stabilized submission that previously produced an official score of 69, we identified and corrected two independent evaluator compatibility issues:
-
A test-environment dependency interaction caused one branch to fail before producing
results.xml. The branchrun.shupgraded pytest at runtime, which produced an incompatibility with the installedlibtmuxpytest plugin. Preventing that unintended pytest upgrade restored normal branch execution. -
JUnit testcase names produced under the evaluator's
eval.tests.*namespace were compared literally against the correspondingtests.*names intests.json. This caused valid executed tests to be simultaneously classified as unexpected and injected asnot_run. We added namespace-aware matching while preserving the existing completeness checks.
We also retained the existing pytest-timeout compatibility handling (thread → signal) already required to prevent xdist worker loss on timed-out tests.
Verification sequence:
- isolated failing branch reproduced;
- minimal evaluator-side correction applied;
- isolated branch: 2/2 passed, no branch errors or warnings;
- evaluator regression suite: 41/41 passed;
- complete ProgramBench official evaluator rerun performed.
Final official result:
wfxr__csview.8ac4de0 ✅ 335 tests
Average 100
The resulting machine-readable evaluation artifact contains:
solution_branch: submissionerror_code: Noneerror_details: None- 348 recorded test results
- 347 passed
- 1 skipped
- 0
not_run - no branch errors
- no warnings
Most importantly, the submission archive was unchanged between the earlier score-69 run and the final solved run.
Submission SHA-256:
a7b055b8dea5dddfadf88d381f634cdf2af3f6ac02b62f024740dd12b3bc512f
Final official eval.json SHA-256:
930daeeb3c4f080a98414b84338e2a6506f64e6e0ef3ab158d77be252a63d2d2
ProgramBench repository HEAD used for the final run:
963063c9271cc40fa179977356782ea4582e0b0c
We preserved the complete evidence package: original submission, final eval.json, precheck reports, reconstruction state, evaluator state, evaluator diff, regression evidence, provenance, manifest, and SHA-256 inventory. The package was independently hash-verified after transfer from the Linux evaluation environment to Windows.
We would be happy to provide the evaluator patch and/or the verification package privately to the ProgramBench maintainers for independent reproduction.
Could you advise the preferred way to submit this result and evaluator finding for official independent verification?
- 主要语言
- Python
- 星标
- 928
- 派生
- 67
- PR 合并指标
- 30 天内没有已合并 PR
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
facebookresearch/ProgramBench 的其他 Issue
-
难度 3/5 1-2 天 新手友好度 68/100
-
难度 4/5 3-5 天 新手友好度 55/100
-
难度 4/5 3-5 天 新手友好度 48/100
-
难度 4/5 3-5 天 新手友好度 58/100
-
难度 5/5 一周以上 新手友好度 25/100
facebookresearch/ProgramBench#50 · 1 条评论 ·
查看 facebookresearch/ProgramBench 的全部 Issue
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 75/100
stephrobert/dsoxlab#238 ·
-
难度 2/5 1-3 小时 新手友好度 75/100
-
难度 2/5 1-3 小时 新手友好度 75/100
sublimehq/package_control#1780 ·
-
难度 2/5 1-3 小时 新手友好度 65/100
-
难度 2/5 1-3 小时 新手友好度 70/100
nwg-piotr/nwg-displays#145 ·