vibrating_path_graph_coloring: 8 cases at once on 4 CPUs turn valid solvers into `inf`

オープン
  • 何か問題がありますか？
#40 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
18/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python
領域
tooling

調査の方向性

The concurrency and timeout behaviour lives in tools/judge_local.py and the task's SCHEMA and timeout_sec settings under edgebench/tasks/vibrating_path_graph_coloring. Start by reproducing the failure: wrap a valid solution.py to spin for 20 s of CPU and run the judge_local command from the issue. Done means the pool size is capped by the CPU count and cases, and the timeout is raised as the proposed fix describes, so valid solvers stop scoring inf. Pull request #36 is referenced in the issue but not linked, and the maintainers still have to decide on regrading old scores.

索引モデルが issue の本文から書いたものです。

説明

Task

vibrating_path_graph_coloring

Claim

The judge runs 8 cases at once on the official 4 CPUs, so a single-threaded case gets about 15 s of CPU (computed) inside its 30 s wall-clock limit. One timeout makes the whole total inf, so valid solvers lose their score, some on every grade and some only on some grades.

Impact

Measured on 8 vCPU hosts with the judge limited to 4 CPUs. Scores are 0..100.

Submission n Current judge 4 cases at once (#36)
Opus 5.5 solver 15 inf every time, 7-39 of 50 OK 19.77-19.98, 50 of 50 OK
Gemini 4 Argon solver 1 3 32.49 twice, inf once 32.49, 50 of 50 OK
Gemini 4 Argon solver 2 3 inf every time, 43-49 of 50 OK 18.64, 50 of 50 OK
Valid answer plus 20 s of CPU per case 3 inf, 2 of 50 OK 2.06, 50 of 50 OK

The Opus file also passed all 8 completed grades during its own run on the published v1 judge image (a ninth stalled, result unknown), so the current grading scores it non-repeatably.

Evidence and how to reproduce

Wrap a valid solution.py so it spins for 20 s of CPU before printing its answer, then grade it:

python3 tools/judge_local.py --task-dir edgebench/tasks/vibrating_path_graph_coloring --image <judge tag> --archive <submission.tar.gz> --out result.json
Proposed fix

#36 sets the pool to min(8, SFORGE_CPUS, cases), as SCHEMA asks, and raises timeout_sec from 600 to 1200. Solvers that read the clock may get different scores, and old scores have no mapping, so maintainers need to decide whether this needs an exception and a regrade.

Limits

We hold no v1 leaderboard submissions, so we can't show how far their scores would move. Known issue 3, general host load, remains: the limit is still wall clock.

We found this during EdgeBench Verified, our audit of EdgeBench's tasks.

主要言語
Python
スター
474
フォーク
20
平均マージ
1日 7時間
マージ済み PR（30日）
2

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ByteDance-Seed/EdgeBench のほかの issue

ByteDance-Seed/EdgeBench の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。