schemathesis_datagen_pipeline: hidden Phase 1 imports names and format keys that TODO.md never gives

Abierto
  • ¿Algo no está bien?
#49 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
testing

Línea de trabajo

Read TODO.md and inspect test_01_header_strategies.py to compare the documented requirements with the hidden Phase 1 imports and checks described in the issue. Reproduce the scores with the provided archive and mk.py using the listed judge_local.py command; done means the documented task requirements allow the same archive to reach the reported full Phase 1 score.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Task

schemathesis_datagen_pipeline

Claim

Hidden Phase 1 (test_01_header_strategies.py) imports names from schemathesis.specs.openapi.headers and checks literal format keys and Range forms that TODO.md doesn't give. Target 1 lists the registry, generators, dispatch and integration it needs, but not the module path, exported names or format keys. A module with other names fails the import, so all of Phase 1 fails: 2 of 13 weights, 15.38 of 100 points.

Impact

Repeated grades of the same archives on judge 484f15fd657f:

Submission n Score Phase 1
Opus 5.5 solver, unchanged 19 72.31 0 of 15
Same, with aliases for the names and keys 15 85.64 (8 times), 86.67 (7 times) 13 or 14 of 15
That plus the three literal Range forms 13 87.69 15 of 15
Evidence and how to reproduce

On community f92857a3, with agent2.tar.gz and mk.py from https://github.com/findalexli/EdgeBench/tree/da540564098c8bbc0ad5b9b4cfce0ce53045f8d6/schemathesis_datagen_pipeline (archive sha256 a8e1ccccf546717cd70cd63d2bd671571193b4c048dcb9a7a96357a1b39a6711):

python -m sforge build --task schemathesis_datagen_pipeline
python3 mk.py schemathesis_datagen_pipeline . agent2_named agent2_full
for a in agent2 agent2_named agent2_full; do
  python3 tools/judge_local.py --task-dir edgebench/tasks/schemathesis_datagen_pipeline \
    --image edgebench.judge.schemathesis_datagen_pipeline:484f15fd657f --archive $a.tar.gz --out $a.json
done

Expected: 72.31, then 85.64 or 86.67, then 87.69.

Limits

All rows come from one Opus run; eight of its snapshots contain a headers.py.

Lenguaje dominante
Python
Estrellas
474
Forks
20
Merge medio
3 d 6 h
PR fusionados (30 d)
3

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ByteDance-Seed/EdgeBench

Todos los issues de ByteDance-Seed/EdgeBench

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.