[Feature]: `graphify benchmark` measures answer size but never whether the answer is right
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
Research direction
Read the existing graphify benchmark command and the query_graph path it uses; the issue does not name source files or tests. Start by tracing how the benchmark invokes queries and formats its report, then compare that path with the proposed --questions, --json, --depth, and --budget options. Done means JSON and JSONL question files produce the requested file-rank metrics and answer sizes while the benchmark without --questions stays unchanged.
Written by the indexing model from the issue text.
Description
Problem or use case
graphify benchmark reports how many tokens a query costs versus reading the corpus. Nothing measures whether the answer contains what the question needed, so a change to query scoring, seeding or rendering can make answers smaller and worse and the benchmark calls it a win. Twenty-odd open PRs change how query ranks or renders (#2403, #3284, #3417, #3636, ...), and there is no shared number to compare them by.
I built such a number from a public set with known answers: the 177 SWE-bench Lite issues of django, pytest, sphinx, requests, flask, pylint, seaborn, xarray and astropy, each repo built at the commit its issues were filed against, the issue title as the question and the files the real fix changed as the right answer. query_graph with the MCP defaults (BFS, depth 3, ~2000 tokens), graphify 0.9.85:
| right file is the first file shown | in the first 3 | in the first 5 | anywhere in the answer | |
|---|---|---|---|---|
query_graph |
25% | 34% | 38% | 62% |
| BM25 over the repo's .py files (reference) | 32% | 56% | 64% | 92% in the top 50 |
The same set run against some open query PRs (each applied on its own base, same graphs):
| first 5 | answers carrying "Complete answer over budget" (full issue text as the question) | |
|---|---|---|
| 0.9.85 | 38% | 55 of 177 |
| #2403 (no test-path seeds) | 51% | 38 |
| #3417 (ranking rewrite) | 51% | 146 |
| #3284 | 39% | 61 |
| #3636 | 38% | 55 |
Proposed solution
graphify benchmark --questions FILE: a JSON list or JSON Lines of {"question": ..., "files": [...]}. Each question goes through query_graph as the MCP tool runs it; the report gives how often an expected file is the first file shown, in the first 3 and 5 files, anywhere, and the answer size. --json for raw per-question results, --depth / --budget to match other settings. A project's own history supplies questions: issue titles and the files their fixes changed.
Alternatives considered
A separate harness outside the repo (what produced the tables above): nobody running or reviewing a query change has it, and it drifts from the real query_graph path.
Area
Graph building or querying
Are you willing to submit a PR?
Yes, I can work on this
Compatibility / behavior impact
CLI and docs only. graphify benchmark without --questions is unchanged; no graph, cache, extraction or MCP output change.
Additional context
Questions built from SWE-bench Lite (problem statement title, files of the gold patch), each repo checked out at the most common base commit of its instances. PR right after this issue.
- Dominant language
- Python
- Stars
- 124k
- Forks
- 11.9k
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Graphify-Labs/graphify
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Graphify-Labs/graphify#3763 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Graphify-Labs/graphify#3611 · 1 comment ·
Maintainers usually reply within 1 day
-
Nix supportOpen
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Graphify-Labs/graphify#3193 · 2 reactions ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
Graphify-Labs/graphify#2871 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Graphify-Labs/graphify#2865 ·
Maintainers usually reply within 1 day
All issues in Graphify-Labs/graphify
Similar issues
-
SSM GetParameter throttling during concurrent workflow launchesPossibly taken @alexiswl claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
-
[Bug] Dev image build fails: `npm install` in the root Dockerfile runs out of memory (exit 134)Possibly taken @andest01 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Rename FunctionalRole BOT_DEV to DEV_STAFFPossibly taken A pull request linked to this issue is open or already merged. Openenhancement good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
pesu-dev/discord_bot#97 ·
-
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
gradio-app/gradio#13964 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 93/100
BSData/horus-heresy-3rd-edition#3232 · 1 comment ·
Maintainers usually reply within 1 day