[Feature]: `graphify benchmark` measures answer size but never whether the answer is right
Los mantenedores suelen responder en 1 día
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 48/100
Línea de trabajo
Read the existing graphify benchmark command and the query_graph path it uses; the issue does not name source files or tests. Start by tracing how the benchmark invokes queries and formats its report, then compare that path with the proposed --questions, --json, --depth, and --budget options. Done means JSON and JSONL question files produce the requested file-rank metrics and answer sizes while the benchmark without --questions stays unchanged.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Problem or use case
graphify benchmark reports how many tokens a query costs versus reading the corpus. Nothing measures whether the answer contains what the question needed, so a change to query scoring, seeding or rendering can make answers smaller and worse and the benchmark calls it a win. Twenty-odd open PRs change how query ranks or renders (#2403, #3284, #3417, #3636, ...), and there is no shared number to compare them by.
I built such a number from a public set with known answers: the 177 SWE-bench Lite issues of django, pytest, sphinx, requests, flask, pylint, seaborn, xarray and astropy, each repo built at the commit its issues were filed against, the issue title as the question and the files the real fix changed as the right answer. query_graph with the MCP defaults (BFS, depth 3, ~2000 tokens), graphify 0.9.85:
| right file is the first file shown | in the first 3 | in the first 5 | anywhere in the answer | |
|---|---|---|---|---|
query_graph |
25% | 34% | 38% | 62% |
| BM25 over the repo's .py files (reference) | 32% | 56% | 64% | 92% in the top 50 |
The same set run against some open query PRs (each applied on its own base, same graphs):
| first 5 | answers carrying "Complete answer over budget" (full issue text as the question) | |
|---|---|---|
| 0.9.85 | 38% | 55 of 177 |
| #2403 (no test-path seeds) | 51% | 38 |
| #3417 (ranking rewrite) | 51% | 146 |
| #3284 | 39% | 61 |
| #3636 | 38% | 55 |
Proposed solution
graphify benchmark --questions FILE: a JSON list or JSON Lines of {"question": ..., "files": [...]}. Each question goes through query_graph as the MCP tool runs it; the report gives how often an expected file is the first file shown, in the first 3 and 5 files, anywhere, and the answer size. --json for raw per-question results, --depth / --budget to match other settings. A project's own history supplies questions: issue titles and the files their fixes changed.
Alternatives considered
A separate harness outside the repo (what produced the tables above): nobody running or reviewing a query change has it, and it drifts from the real query_graph path.
Area
Graph building or querying
Are you willing to submit a PR?
Yes, I can work on this
Compatibility / behavior impact
CLI and docs only. graphify benchmark without --questions is unchanged; no graph, cache, extraction or MCP output change.
Additional context
Questions built from SWE-bench Lite (problem statement title, files of the gold patch), each repo checked out at the most common base commit of its instances. PR right after this issue.
- Lenguaje dominante
- Python
- Estrellas
- 124k
- Forks
- 11.9k
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de Graphify-Labs/graphify
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
Graphify-Labs/graphify#3763 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Graphify-Labs/graphify#3611 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Nix supportAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
Graphify-Labs/graphify#3193 · 2 reacciones ·
Los mantenedores suelen responder en 1 día
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 92/100
Graphify-Labs/graphify#2871 · 2 comentarios ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
Graphify-Labs/graphify#2865 ·
Los mantenedores suelen responder en 1 día
Todos los issues de Graphify-Labs/graphify
Issues similares
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
jbacalso24/mimry#291 ·
Los mantenedores suelen responder en 1 día
-
good first issue
Dificultad 2/5 Menos de una hora Aptitud para principiantes 82/100
eli-alvarado/us-watchlist-alerts#14 ·
Los mantenedores suelen responder en 1 día
-
from-review-extraction python severity:nit
Dificultad 2/5 1-3 horas Aptitud para principiantes 80/100
LearningCircuit/local-deep-research#7462 ·
Los mantenedores suelen responder en 2 días
-
mermaid_extract.py silently discards single-line state-diagram notes; sequence notes are keptAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
cathrynlavery/diagram-design#441 ·
Los mantenedores suelen responder en 2 días
-
[Cleanup] Include the public header in work_split.cppPosiblemente ocupada @blozano-tt la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
tenstorrent/tt-metal#60850 ·
Los mantenedores suelen responder en 1 día