Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

README benchmark: what changes when the baseline already has repo rules

Abierto Apto para principiantes
  • ¿Algo no está bien?
#1,700 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
72/100
Tipo de issue
Documentación
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
markdown
Área
documentation

Línea de trabajo

Comienza en la tabla de benchmarks del README que compara codegraph con Claude Code con una configuración MCP vacía, y lee la tabla del issue #1700 con los valores de línea base de los rules-on (5.67 → 3.33 llamadas, $0.51 → $0.59 por tarea). Se considera completado cuando la README muestre una segunda fila de benchmark o una nota para los repos cuyo CLAUDE.md ya exige Grep-then-Read, con la advertencia del reportero de que el coste aumenta ligeramente mientras las llamadas disminuyen. Confirma con el mantenedor si las cifras reportadas pueden publicarse antes de editar.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

The README's benchmark compares codegraph against Claude Code with an empty MCP config and no instruction files. Most repos that would install codegraph already carry a CLAUDE.md or equivalent, and the one I measured on tells the agent to Grep -n first and then Read with offset/limit. Against that baseline the tool still cuts calls, but by less than the README's percentages, and it costs a little more per task rather than less. Offered as a data point for the README, not a complaint; the tool stays installed here.

Setup

Headless Claude Code (claude -p, Opus, --effort medium, --permission-mode dontAsk), one fresh session per cell, a private config dir per arm, on a ~500-file TypeScript/Vue Chrome-extension repo. Tasks are code-discovery questions with a graded answer (which function owns X, what calls Y, trace Z across files). Pairs are B − A by task; p is a paired sign test.

Arm n tool calls / task cost / task correct
no rules, no codegraph (the README's baseline) 6 8.00 $0.35 5/6
no rules + codegraph 6 3.67 $0.38 6/6
repo rules, no codegraph 12 5.67 $0.51 12/12
repo rules + codegraph (eager, no prompt hook) 12 3.33 $0.59 12/12

With rules on, calls drop by a median 2 per task (p .015) and cost rises by a median $0.13 (p .18). The cost goes up because an explore answer averaged ~21.7k characters here against ~4.6k for a bounded Read, so the context each later turn re-reads is larger even though there are fewer turns. Without rules the tool's call saving is bigger (8 → 3.67) because the baseline is worse, which is the README's row; the rules alone already take the baseline from 8 to 5.67.

The same shape held in every framing I tried: with the vendor prompt hook installed, and with the tool deferred behind Claude Code's tool search (#1696). −2 to −2.5 calls per task, +$0.08–0.15 per cell, correctness unchanged, never under p .05 at n = 6.

What decided adoption

The model used the tool in 4 of 6 discovery tasks no matter how it was loaded or instructed. The two it skipped anchor on a storage-key string and a CLI flag, and on those a query returns padding (45 and 132 symbols, none of them the answer), so Grep is the right tool there. Prompts that name a concept and no identifier went the other way: 3 of 3 adopted, 3 of 3 correct against 2 of 3 for Grep-and-Read, at 2.3 calls against 6.3. So the trigger that held up in the repo rules is the shape of the question: an identifier, a call path, or a concept → one explore call; a string literal, flag, or regex → Grep.

In daily use

Over three days of interactive sessions on the same repo (168 transcripts, mostly Fable), the audit counts 29 explore calls against 57 serial Grep-then-Read chains anchored on a code symbol, the chains the tool is meant to replace; on the day the rules were flipped to send markdown questions to it as well, 20 calls in 80 transcripts, about one per chain. So the headless adoption ceiling carries over: the tool replaces roughly half of the chains it could, and the half it misses is habit on string-shaped questions, not loading.

Suggestion

A README row with a rules-on baseline (a CLAUDE.md that already says "grep, then read a range") would tell a prospective user what to expect on their own repo better than the empty-config row does. Happy to share the harness, the task bank, and the per-cell transcripts.

Lenguaje dominante
C
Estrellas
73.2k
Forks
4.7k
Merge medio
2 h 13 min
PR fusionados (30 d)
418

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de colbymchenry/codegraph

Todos los issues de colbymchenry/codegraph

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.