Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Drop the 1024 Gecko file: chunk long notes on the 256 file, measure first

Abierto
  • ¿Algo no está bien?
#296 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
dart, flutter, sqlite
Área
databases, mobile, search

Línea de trabajo

Start with docs/SHIVA/embedding.md and the named entry points: NoteVectorCleanupWorker.sweepOrphans, purgeNotesInTxn, TostoreVectorRepositoryImpl.search, ExtractKnowledgeUseCase, NoteEmbeddingWorker, and EmbeddingService. Implement stable note piece IDs and handle cleanup, deletion, search deduplication, extraction, Manas scope, and interrupted embedding; add the three planned tests. Re-measure on the phone against the listed whole-note results, and keep the change only if long-note and short-note search results stay as good.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement good first issue P2-Medium

Goal

Ship only the 256-token Gecko file and drop the 1024-token one: the app gets 145 MB smaller (31 MB smaller than before #295), and every embed takes ~2.2 s instead of up to 11.7 s on a weak phone.

The blocker is long notes. The 256 file reads about 1,100 characters, so a longer note would be cut and its later text would not be searchable. To drop the 1024 file, notes over ~1,100 characters must be chunked: several vectors per note, each piece embedded with the 256 file.

Where things stand (#295)

  • The app bundles both files. It embeds with 256 and switches to 1024 only for text that does not fit (~1.1 s a switch).
  • Both files give the same vector for the same text (cosine 1.0000), so old and new vectors share one index. Vectors made by the 1024 file stay valid after it is removed.
  • PDF/DOCX chunks (at most 700 characters) already fit the 256 file.

What was measured (vivo 1933, see docs/SHIVA/embedding.md)

14 long notes (950-2,000 characters), 16 short notes in the same index, and 44 questions (28 aimed at the start or the end of a long note):

Long-note questions: first / top 3 Short-note questions: first / top 3 One note fills 2+ of the top 3 Time to index
Whole note, 1024 file 25/28 · 28/28 13/16 · 16/16 0/44 352 s
700-character chunks, 256 file 27/28 · 28/28 12/16 · 15/16 32/44 109 s
  • Better: questions about a long note found it first more often (27/28 against 25/28).
  • Worse: one short note dropped out of the top 3, because small pieces of long notes compete harder with short notes.
  • Crowding: one note often filled several of the top results, so search must keep only the best piece per note.

What chunking a note changes

Place Today With pieces If not handled
Launch sweep (NoteVectorCleanupWorker.sweepOrphans) deletes any vector whose id is not a note id a piece id (noteId#2) is not a note id every piece is deleted on each launch. Must be done first
Delete queue (purgeNotesInTxn → PendingVectorDelete) deletes the vector noteId must delete every piece pieces of deleted notes stay searchable
Search (TostoreVectorRepositoryImpl.search) vector id = note id map pieces back to the note, keep the best piece one note fills the top results
Small-model cut-off (0.72) calibrated on whole notes and PDF chunks piece scores may differ re-measure with the calibration tests
Knowledge extraction (ExtractKnowledgeUseCase) one vector per note to find similar notes, self-excluded by id which vector, and exclude the note's own pieces the note finds itself as "similar"
Manas scope (#289) filters by note ids must include piece ids long notes vanish from a Manas chat
Embedding worker (NoteEmbeddingWorker) one embed per note split, embed each piece, remove the queue row after the last one half-embedded notes after a kill (fine if piece ids are stable)

Plan

  1. Build chunking for notes over ~1,100 characters on the 256 file, with every row above handled, and tests for: deleting a note removes all its pieces; the launch sweep keeps pieces of live notes; search returns each note once.
  2. Re-measure on the phone with the same set as above (plus some notes over 4,500 characters, which the 1024 file cuts today): right note first and top 3 for long-note and short-note questions, crowding, time, and the cut-off calibration.
  3. Keep it only if the results stay as good as whole notes on the 1024 file, short-note questions included. Then remove gecko_1024_quant.tflite and the switching code in EmbeddingService.
  4. If it is not as good, keep #295 as it is and close this.

Refs #266, #295.

Lenguaje dominante
Dart
Estrellas
2
Forks
2
Merge medio
1 d 7 h
PR fusionados (30 d)
35

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de basictech01/uniun

Todos los issues de basictech01/uniun

Issues similares

Más issues de Dart

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.