Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

[Feature]: `graphify benchmark` measures answer size but never whether the answer is right

オープン
  • 何か問題がありますか？
#4,341 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@rohit-jsfreaky がすでに取り組んでいます。

2026年10月11日 から。

  • #4342 @rohit-jsfreaky による — オープン

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python
領域
cli

調査の方向性

Read the existing graphify benchmark command and the query_graph path it uses; the issue does not name source files or tests. Start by tracing how the benchmark invokes queries and formats its report, then compare that path with the proposed --questions, --json, --depth, and --budget options. Done means JSON and JSONL question files produce the requested file-rank metrics and answer sizes while the benchmark without --questions stays unchanged.

索引モデルが issue の本文から書いたものです。

説明

enhancement
Problem or use case

graphify benchmark reports how many tokens a query costs versus reading the corpus. Nothing measures whether the answer contains what the question needed, so a change to query scoring, seeding or rendering can make answers smaller and worse and the benchmark calls it a win. Twenty-odd open PRs change how query ranks or renders (#2403, #3284, #3417, #3636, ...), and there is no shared number to compare them by.

I built such a number from a public set with known answers: the 177 SWE-bench Lite issues of django, pytest, sphinx, requests, flask, pylint, seaborn, xarray and astropy, each repo built at the commit its issues were filed against, the issue title as the question and the files the real fix changed as the right answer. query_graph with the MCP defaults (BFS, depth 3, ~2000 tokens), graphify 0.9.85:

right file is the first file shown in the first 3 in the first 5 anywhere in the answer
query_graph 25% 34% 38% 62%
BM25 over the repo's .py files (reference) 32% 56% 64% 92% in the top 50

The same set run against some open query PRs (each applied on its own base, same graphs):

first 5 answers carrying "Complete answer over budget" (full issue text as the question)
0.9.85 38% 55 of 177
#2403 (no test-path seeds) 51% 38
#3417 (ranking rewrite) 51% 146
#3284 39% 61
#3636 38% 55
Proposed solution

graphify benchmark --questions FILE: a JSON list or JSON Lines of {"question": ..., "files": [...]}. Each question goes through query_graph as the MCP tool runs it; the report gives how often an expected file is the first file shown, in the first 3 and 5 files, anywhere, and the answer size. --json for raw per-question results, --depth / --budget to match other settings. A project's own history supplies questions: issue titles and the files their fixes changed.

Alternatives considered

A separate harness outside the repo (what produced the tables above): nobody running or reviewing a query change has it, and it drifts from the real query_graph path.

Area

Graph building or querying

Are you willing to submit a PR?

Yes, I can work on this

Compatibility / behavior impact

CLI and docs only. graphify benchmark without --questions is unchanged; no graph, cache, extraction or MCP output change.

Additional context

Questions built from SWE-bench Lite (problem statement title, files of the gold patch), each repo checked out at the most common base commit of its instances. PR right after this issue.

主要言語
Python
スター
124k
フォーク
11.9k
30日以内にマージされた PR はありません

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

Graphify-Labs/graphify のほかの issue

Graphify-Labs/graphify の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。