Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

benchmarks: code-intelligence result inaccessible; BENCHMARKS.md is a 404; headline metrics test conversation memory, not code graphs

Cerrado
  • ¿Algo no está bien?
#3,690 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
42/100
Tipo de issue
Documentación
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
fastapi, python
Área
documentation, testing

Línea de trabajo

Start with the README benchmarks table and its BENCHMARKS.md link, then check whether BENCHMARKS.md exists on main and review the references to #1706 and #2270. Confirm where the code-intelligence result and per-system tables are maintained. Done means restoring or correcting the link and documenting the requested INFERRED-edge precision result for the FastAPI demo corpus.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

help wanted

The README benchmarks table shows three results — LOCOMO recall@10, LOCOMO QA accuracy, and LongMemEval-S QA accuracy — and directs to BENCHMARKS.md for "full per-system tables, the code-intelligence result, and reproduction commands." BENCHMARKS.md returns 404 on main.

Two problems follow from this:

1. The only code-relevant benchmark is unreachable

graphify's primary pitch is code knowledge graphs — AST parsing, cross-file link resolution, EXTRACTED vs INFERRED confidence tags. LOCOMO and LongMemEval benchmark conversational memory retrieval over chat transcripts. A user evaluating graphify for code navigation has no published number to look at. The code-intelligence result — presumably the metric that matters most for this tool's stated purpose — exists only in the missing file.

2. INFERRED edge precision for code graphs is unmeasured

The headline differentiator is the EXTRACTED / INFERRED tag: "you can tell what was read directly from what was inferred." The LOCOMO and LongMemEval results say nothing about whether INFERRED cross-file links in code are correct. Per #2270, confidence: EXTRACTED already appears on semantically-derived nodes; the complementary question — what fraction of INFERRED edges from AST resolution are accurate — has no number attached to it anywhere.

Suggested next steps

  • Either commit BENCHMARKS.md to main or update the README link to wherever the code-intelligence result and per-system tables live. (#1706 notes the memory harness is a separate repo; the code-intelligence half presumably is too — a link would suffice.)
  • Add a precision figure for INFERRED edges on the FastAPI demo corpus shown in the README. A 100-edge manual sample against ground-truth cross-file references would make the confidence-tag claim checkable without a full eval pipeline.

Happy to help with either.

Lenguaje dominante
Python
Estrellas
124k
Forks
11.9k
Sin PR fusionados en 30 d

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Graphify-Labs/graphify

Todos los issues de Graphify-Labs/graphify

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.