Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

README benchmark: what changes when the baseline already has repo rules

Open Beginner friendly
  • Is something wrong?
#1,700 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Quiet
Tech stack
markdown
Domain
documentation

Research direction

Start with the README benchmark section mentioned in the issue and compare its existing baseline row with the reported rules-on results. Add a row or comparison for the repo-rules baseline and codegraph arm, making clear that the calls decrease while cost rises in this setup. Done when readers can distinguish the empty-config results from the rules-on results and understand the reported tradeoff.

Written by the indexing model from the issue text.

Description

Summary

The README's benchmark compares codegraph against Claude Code with an empty MCP config and no instruction files. Most repos that would install codegraph already carry a CLAUDE.md or equivalent, and the one I measured on tells the agent to Grep -n first and then Read with offset/limit. Against that baseline the tool still cuts calls, but by less than the README's percentages, and it costs a little more per task rather than less. Offered as a data point for the README, not a complaint; the tool stays installed here.

Setup

Headless Claude Code (claude -p, Opus, --effort medium, --permission-mode dontAsk), one fresh session per cell, a private config dir per arm, on a ~500-file TypeScript/Vue Chrome-extension repo. Tasks are code-discovery questions with a graded answer (which function owns X, what calls Y, trace Z across files). Pairs are B − A by task; p is a paired sign test.

Arm n tool calls / task cost / task correct
no rules, no codegraph (the README's baseline) 6 8.00 $0.35 5/6
no rules + codegraph 6 3.67 $0.38 6/6
repo rules, no codegraph 12 5.67 $0.51 12/12
repo rules + codegraph (eager, no prompt hook) 12 3.33 $0.59 12/12

With rules on, calls drop by a median 2 per task (p .015) and cost rises by a median $0.13 (p .18). The cost goes up because an explore answer averaged ~21.7k characters here against ~4.6k for a bounded Read, so the context each later turn re-reads is larger even though there are fewer turns. Without rules the tool's call saving is bigger (8 → 3.67) because the baseline is worse, which is the README's row; the rules alone already take the baseline from 8 to 5.67.

The same shape held in every framing I tried: with the vendor prompt hook installed, and with the tool deferred behind Claude Code's tool search (#1696). −2 to −2.5 calls per task, +$0.08–0.15 per cell, correctness unchanged, never under p .05 at n = 6.

What decided adoption

The model used the tool in 4 of 6 discovery tasks no matter how it was loaded or instructed. The two it skipped anchor on a storage-key string and a CLI flag, and on those a query returns padding (45 and 132 symbols, none of them the answer), so Grep is the right tool there. Prompts that name a concept and no identifier went the other way: 3 of 3 adopted, 3 of 3 correct against 2 of 3 for Grep-and-Read, at 2.3 calls against 6.3. So the trigger that held up in the repo rules is the shape of the question: an identifier, a call path, or a concept → one explore call; a string literal, flag, or regex → Grep.

In daily use

Over three days of interactive sessions on the same repo (168 transcripts, mostly Fable), the audit counts 29 explore calls against 57 serial Grep-then-Read chains anchored on a code symbol, the chains the tool is meant to replace; on the day the rules were flipped to send markdown questions to it as well, 20 calls in 80 transcripts, about one per chain. So the headless adoption ceiling carries over: the tool replaces roughly half of the chains it could, and the half it misses is habit on string-shaped questions, not loading.

Suggestion

A README row with a rules-on baseline (a CLAUDE.md that already says "grep, then read a range") would tell a prospective user what to expect on their own repo better than the empty-config row does. Happy to share the harness, the task bank, and the per-cell transcripts.

Dominant language
C
Stars
73.2k
Forks
4.7k
Avg merge
2h 13m
Merged PRs (30d)
418

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from colbymchenry/codegraph

All issues in colbymchenry/codegraph

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.