vibrating_path_graph_coloring: 8 cases at once on 4 CPUs turn valid solvers into `inf`

Abierto
  • ¿Algo no está bien?
#40 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
18/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
tooling

Línea de trabajo

The concurrency and timeout behaviour lives in tools/judge_local.py and the task's SCHEMA and timeout_sec settings under edgebench/tasks/vibrating_path_graph_coloring. Start by reproducing the failure: wrap a valid solution.py to spin for 20 s of CPU and run the judge_local command from the issue. Done means the pool size is capped by the CPU count and cases, and the timeout is raised as the proposed fix describes, so valid solvers stop scoring inf. Pull request #36 is referenced in the issue but not linked, and the maintainers still have to decide on regrading old scores.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Task

vibrating_path_graph_coloring

Claim

The judge runs 8 cases at once on the official 4 CPUs, so a single-threaded case gets about 15 s of CPU (computed) inside its 30 s wall-clock limit. One timeout makes the whole total inf, so valid solvers lose their score, some on every grade and some only on some grades.

Impact

Measured on 8 vCPU hosts with the judge limited to 4 CPUs. Scores are 0..100.

Submission n Current judge 4 cases at once (#36)
Opus 5.5 solver 15 inf every time, 7-39 of 50 OK 19.77-19.98, 50 of 50 OK
Gemini 4 Argon solver 1 3 32.49 twice, inf once 32.49, 50 of 50 OK
Gemini 4 Argon solver 2 3 inf every time, 43-49 of 50 OK 18.64, 50 of 50 OK
Valid answer plus 20 s of CPU per case 3 inf, 2 of 50 OK 2.06, 50 of 50 OK

The Opus file also passed all 8 completed grades during its own run on the published v1 judge image (a ninth stalled, result unknown), so the current grading scores it non-repeatably.

Evidence and how to reproduce

Wrap a valid solution.py so it spins for 20 s of CPU before printing its answer, then grade it:

python3 tools/judge_local.py --task-dir edgebench/tasks/vibrating_path_graph_coloring --image <judge tag> --archive <submission.tar.gz> --out result.json
Proposed fix

#36 sets the pool to min(8, SFORGE_CPUS, cases), as SCHEMA asks, and raises timeout_sec from 600 to 1200. Solvers that read the clock may get different scores, and old scores have no mapping, so maintainers need to decide whether this needs an exception and a regrade.

Limits

We hold no v1 leaderboard submissions, so we can't show how far their scores would move. Known issue 3, general host load, remains: the limit is still wall clock.

We found this during EdgeBench Verified, our audit of EdgeBench's tasks.

Lenguaje dominante
Python
Estrellas
474
Forks
20
Merge medio
3 d 6 h
PR fusionados (30 d)
3

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ByteDance-Seed/EdgeBench

Todos los issues de ByteDance-Seed/EdgeBench

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.