Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

sgm-index: re-embed only new or changed notes, with a --full rebuild flag

Aperta
  • Qualcosa non va?
#17 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
1-2 giorni
Idoneità per principianti
62/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, sqlite
Ambito
backend, cli

Direzione di ricerca

Start with main() in src/sqlite_graph_memory/index_notes.py, which loops over iter_notes() and chunks_of(), then read brain_ask.load_index() to confirm the on-disk format that must not change. Extract a build_index(notes_dir, out_dir, encoder=None, full=False) function and add a manifest keyed by note path. Done when a counting fake-encoder test in tests/test_incremental_index.py shows zero texts embedded on an unchanged rerun, and the .npy rows and meta list stay aligned after a delete.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

accepted help wanted

Drafted by Mycroft, the lab's synthetic AI co-founder; reviewed by @tonydzi.

Goal. Make sgm-index re-embed only the notes that are new or changed since the last run, drop the ones that were deleted, and keep a --full flag for a complete rebuild.

Why it matters. Today the indexer reads every note and re-embeds every chunk on each run (a 24k-chunk vault takes minutes on a GPU and much longer on CPU). That cost is the reason people index less often than they should, and the README lists incremental indexing as "intentionally missing". A stale index quietly returns old content, so cheaper runs mean fresher recall.

Where.

  • src/sqlite_graph_memory/index_notes.py (main()): today it loops over iter_notes(), chunks with chunks_of(), encodes everything, and overwrites _brain_e5.npy and _brain_e5_meta.pkl. Split the work into a function you can call from a test, for example build_index(notes_dir, out_dir, encoder=None, full=False), where encoder is injectable so tests never load a model. Keep main() as the CLI wrapper.
  • Same file: store a small manifest next to the index (for example _brain_e5_manifest.json) mapping each note path to its size, mtime and a content hash, plus a fingerprint of the settings that change chunk content (E5_MODEL, CHUNK_CHARS, CHUNK_OVERLAP, SNIPPET_CHARS). If the fingerprint differs, do a full rebuild. Unchanged notes keep their existing rows in the .npy matrix and the meta list; new or changed notes are chunked and encoded; deleted notes' rows are removed. The two files must stay row-aligned, and the output format (matrix of float32 rows, list of dicts with path, title, snippet, date) must not change, because brain_ask.load_index() reads it.
  • Same file: add an argparse flag --full to force a rebuild; keep sgm-index <notes_dir> working as before.
  • tests/test_index_notes.py (or a new tests/test_incremental_index.py): add the tests below with a fake encoder that counts how many texts it was asked to embed.
  • README.md: remove "No incremental indexing" from "What's intentionally missing" and document --full.

How to check.

pip install numpy pytest
python -m pytest -q tests/test_incremental_index.py
python -m pytest -q

Expected: the new tests pass and the whole suite stays green (baseline: 88 passed). The key test: index a temp folder of 3 notes with the fake encoder (it records every text it embeds); run again unchanged and assert the encoder saw zero texts; edit one note and assert only that note's chunks were embedded; delete one and assert its rows are gone and the matrix and meta lengths still match. Show that test failing first against a build that ignores the manifest. A manual run with a real model (pip install -e '.[embeddings]', then sgm-index <folder> twice) is a nice extra but not required.

Done when

  • a second run over an unchanged folder embeds nothing, and a test proves it with a counting fake encoder
  • changed and new notes are re-embedded; deleted notes are removed; .npy rows and meta stay aligned
  • --full rebuilds everything; a changed chunk setting or model name triggers a full rebuild automatically
  • the on-disk index format read by brain_ask.load_index() is unchanged

Size. ~1 day

Ask here. https://github.com/tonydzi/sqlite-graph-memory/discussions, or comment on this issue.

Claim it by commenting "claiming this" — no permission needed, and it is yours for 7 days.
You keep the copyright to your code. No CLA, no assignment, ever. We answer every issue and PR within 48 hours, including "no, and here is why" — our silence is our bug, so ping the thread.

Full deal: CONTRIBUTING.md

Lingua principale
Python
Stelle
9
Fork
6
Merge medio
20h 2m
PR unite (30g)
14

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tonydzi/sqlite-graph-memory

Tutte le issue di tonydzi/sqlite-graph-memory

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.