Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Add eval/run_ablation.py: sweep graph-expansion settings and report per query class

Aperta
  • Qualcosa non va?
#19 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
50/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, sqlite
Ambito
search, testing

Direzione di ricerca

Start by reading eval/run_eval.py (score_gold(), recall_at, mrr_at, ndcg_at) and the pipeline functions in src/sqlite_graph_memory/brain_ask.py (query_sims, retrieve_candidates, expand_1hop, rerank_candidates, looks_like_entity). Build a new eval/run_ablation.py that loops over the grid and calls those functions unchanged, computing query_sims once per question. Done when --list-grid prints every cell without loading a model, the CSV has one row per cell and class with n shown, and python -m pytest -q still passes with new tests for grid enumeration and aggregation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

accepted help wanted

Drafted by Mycroft, the lab's synthetic AI co-founder; reviewed by @tonydzi.

Goal. Add eval/run_ablation.py: a sweep over the graph-expansion settings that scores every combination with the existing metrics and writes one results table, per question class. The point is to answer "does the graph help, for which kinds of query, at which settings" with numbers instead of one average.

Why it matters. The retrieval pipeline has five knobs that interact (below), and the repo has only ever scored two settings: vector-only and the shipped graph setting. Anyone tuning a memory layer over their own notes needs to know which knobs matter for which query class (title lookup, body lookup, "bridge" questions whose answers are linked notes, temporal questions). Other people's corpora will disagree with ours, so the runner has to work on any markdown folder.

Where.

  • eval/run_ablation.py: NEW. Reuse the pipeline functions exactly as eval/run_eval.py (score_gold()) does, so the sweep measures the code that ships: ba.query_sims(), ba.retrieve_candidates(sims, meta, topk=...), ba.expand_1hop(sims, meta, base, by_base, ghops=..., gmax=...), ba.rerank_candidates(ce, query, cand, meta, topn=...), with ba = sqlite_graph_memory.brain_ask. Load the encoder and reranker once; compute query_sims once per question; only the candidate-set and rerank steps vary per cell.
  • The five axes, with defaults in src/sqlite_graph_memory/brain_ask.py (TOPK_RETRIEVE = 60, GHOPS, GMAX = 15, 40, looks_like_entity()): (1) hops: 0 (vector only) and 1 (what expand_1hop does; deeper hops do not exist in the code yet, so treat 2+ as a follow-up issue, not part of this one); (2) seed cap ghops (how many top hits are expanded), e.g. 5, 15, 30; (3) neighbour cap gmax, e.g. 10, 40, 80; (4) rerank pool topk (candidates entering the reranker), e.g. 30, 60, 100; (5) gating: entity-like queries (looks_like_entity(query)) get the graph switched off, on or off for the sweep. Keep the default grid small enough to finish in a reasonable time and let --grid override it.
  • Write one results table (CSV and a Markdown view) with columns: cell settings, class, n, Recall@12, MRR, nDCG@12, plus a column for the difference against the vector-only cell. Reuse recall_at, mrr_at, ndcg_at from eval/run_eval.py.
  • README.md: add a short "Ablation" section under the eval chapter with the command and, once you have run it, the table from the mini-vault or your own notes folder.
  • tests/test_eval.py (or a new file): test the grid enumeration and the per-cell aggregation with no models.

How to check.

pip install -e '.[embeddings]'
python eval/build_gold.py <folder-of-markdown> --n-per-class 60     # after: sgm-index <folder>
python eval/run_ablation.py --list-grid                              # prints the cells, loads no model
python eval/run_ablation.py --gold eval/gold-<date>.jsonl --out ablation.csv
python -m pytest -q

Expected: --list-grid prints every combination and its count; the full run writes ablation.csv with one row per (cell, class). Baseline today: eval/run_ablation.py does not exist; python eval/run_eval.py --selftest prints SELFTEST PASS and python -m pytest -q gives 88 passed. If the synthetic mini-vault from issue #16 has landed, use it as the corpus so results are reproducible; otherwise any folder of markdown notes with [[wikilinks]] works.

Done when

  • eval/run_ablation.py sweeps all five axes (hops limited to 0 and 1) and writes one table with a row per cell and class
  • numbers are reported per class, with n shown and a warning for classes under 20 questions
  • every cell calls the shipped functions in brain_ask.py (no reimplemented pipeline), and the no-model parts have tests
  • the PR includes the table from one real run and a short note on which settings it supports or contradicts (changing the shipped defaults is a separate maintainer decision, made after we re-run on our own data)

Size. ~1 day

Ask here. https://github.com/tonydzi/sqlite-graph-memory/discussions, or comment on this issue.

Claim it by commenting "claiming this" — no permission needed, and it is yours for 7 days.
You keep the copyright to your code. No CLA, no assignment, ever. We answer every issue and PR within 48 hours, including "no, and here is why" — our silence is our bug, so ping the thread.

Full deal: CONTRIBUTING.md

Lingua principale
Python
Stelle
9
Fork
6
Merge medio
20h 2m
PR unite (30g)
14

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tonydzi/sqlite-graph-memory

Tutte le issue di tonydzi/sqlite-graph-memory

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.