Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Drop the 1024 Gecko file: chunk long notes on the 256 file, measure first

Đang mở
  • Có gì không ổn?
#296 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
25/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
dart, flutter, sqlite
Lĩnh vực
databases, mobile, search

Hướng nghiên cứu

Start with docs/SHIVA/embedding.md and the named entry points: NoteVectorCleanupWorker.sweepOrphans, purgeNotesInTxn, TostoreVectorRepositoryImpl.search, ExtractKnowledgeUseCase, NoteEmbeddingWorker, and EmbeddingService. Implement stable note piece IDs and handle cleanup, deletion, search deduplication, extraction, Manas scope, and interrupted embedding; add the three planned tests. Re-measure on the phone against the listed whole-note results, and keep the change only if long-note and short-note search results stay as good.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

enhancement good first issue P2-Medium

Goal

Ship only the 256-token Gecko file and drop the 1024-token one: the app gets 145 MB smaller (31 MB smaller than before #295), and every embed takes ~2.2 s instead of up to 11.7 s on a weak phone.

The blocker is long notes. The 256 file reads about 1,100 characters, so a longer note would be cut and its later text would not be searchable. To drop the 1024 file, notes over ~1,100 characters must be chunked: several vectors per note, each piece embedded with the 256 file.

Where things stand (#295)

  • The app bundles both files. It embeds with 256 and switches to 1024 only for text that does not fit (~1.1 s a switch).
  • Both files give the same vector for the same text (cosine 1.0000), so old and new vectors share one index. Vectors made by the 1024 file stay valid after it is removed.
  • PDF/DOCX chunks (at most 700 characters) already fit the 256 file.

What was measured (vivo 1933, see docs/SHIVA/embedding.md)

14 long notes (950-2,000 characters), 16 short notes in the same index, and 44 questions (28 aimed at the start or the end of a long note):

Long-note questions: first / top 3 Short-note questions: first / top 3 One note fills 2+ of the top 3 Time to index
Whole note, 1024 file 25/28 · 28/28 13/16 · 16/16 0/44 352 s
700-character chunks, 256 file 27/28 · 28/28 12/16 · 15/16 32/44 109 s
  • Better: questions about a long note found it first more often (27/28 against 25/28).
  • Worse: one short note dropped out of the top 3, because small pieces of long notes compete harder with short notes.
  • Crowding: one note often filled several of the top results, so search must keep only the best piece per note.

What chunking a note changes

Place Today With pieces If not handled
Launch sweep (NoteVectorCleanupWorker.sweepOrphans) deletes any vector whose id is not a note id a piece id (noteId#2) is not a note id every piece is deleted on each launch. Must be done first
Delete queue (purgeNotesInTxn → PendingVectorDelete) deletes the vector noteId must delete every piece pieces of deleted notes stay searchable
Search (TostoreVectorRepositoryImpl.search) vector id = note id map pieces back to the note, keep the best piece one note fills the top results
Small-model cut-off (0.72) calibrated on whole notes and PDF chunks piece scores may differ re-measure with the calibration tests
Knowledge extraction (ExtractKnowledgeUseCase) one vector per note to find similar notes, self-excluded by id which vector, and exclude the note's own pieces the note finds itself as "similar"
Manas scope (#289) filters by note ids must include piece ids long notes vanish from a Manas chat
Embedding worker (NoteEmbeddingWorker) one embed per note split, embed each piece, remove the queue row after the last one half-embedded notes after a kill (fine if piece ids are stable)

Plan

  1. Build chunking for notes over ~1,100 characters on the 256 file, with every row above handled, and tests for: deleting a note removes all its pieces; the launch sweep keeps pieces of live notes; search returns each note once.
  2. Re-measure on the phone with the same set as above (plus some notes over 4,500 characters, which the 1024 file cuts today): right note first and top 3 for long-note and short-note questions, crowding, time, and the cut-off calibration.
  3. Keep it only if the results stay as good as whole notes on the 1024 file, short-note questions included. Then remove gecko_1024_quant.tflite and the switching code in EmbeddingService.
  4. If it is not as good, keep #295 as it is and close this.

Refs #266, #295.

Ngôn ngữ chính
Dart
Star
2
Fork
2
Merge trung bình
1 ngày 35 phút
Pull request đã merge (30 ngày)
36

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của basictech01/uniun

Tất cả issue của basictech01/uniun

Issue tương tự

Thêm issue về Dart

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.