Drop the 1024 Gecko file: chunk long notes on the 256 file, measure first
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 25/100
調査の方向性
Start with docs/SHIVA/embedding.md and the named entry points: NoteVectorCleanupWorker.sweepOrphans, purgeNotesInTxn, TostoreVectorRepositoryImpl.search, ExtractKnowledgeUseCase, NoteEmbeddingWorker, and EmbeddingService. Implement stable note piece IDs and handle cleanup, deletion, search deduplication, extraction, Manas scope, and interrupted embedding; add the three planned tests. Re-measure on the phone against the listed whole-note results, and keep the change only if long-note and short-note search results stay as good.
索引モデルが issue の本文から書いたものです。
説明
Goal
Ship only the 256-token Gecko file and drop the 1024-token one: the app gets 145 MB smaller (31 MB smaller than before #295), and every embed takes ~2.2 s instead of up to 11.7 s on a weak phone.
The blocker is long notes. The 256 file reads about 1,100 characters, so a longer note would be cut and its later text would not be searchable. To drop the 1024 file, notes over ~1,100 characters must be chunked: several vectors per note, each piece embedded with the 256 file.
Where things stand (#295)
- The app bundles both files. It embeds with 256 and switches to 1024 only for text that does not fit (~1.1 s a switch).
- Both files give the same vector for the same text (cosine 1.0000), so old and new vectors share one index. Vectors made by the 1024 file stay valid after it is removed.
- PDF/DOCX chunks (at most 700 characters) already fit the 256 file.
What was measured (vivo 1933, see docs/SHIVA/embedding.md)
14 long notes (950-2,000 characters), 16 short notes in the same index, and 44 questions (28 aimed at the start or the end of a long note):
| Long-note questions: first / top 3 | Short-note questions: first / top 3 | One note fills 2+ of the top 3 | Time to index | |
|---|---|---|---|---|
| Whole note, 1024 file | 25/28 · 28/28 | 13/16 · 16/16 | 0/44 | 352 s |
| 700-character chunks, 256 file | 27/28 · 28/28 | 12/16 · 15/16 | 32/44 | 109 s |
- Better: questions about a long note found it first more often (27/28 against 25/28).
- Worse: one short note dropped out of the top 3, because small pieces of long notes compete harder with short notes.
- Crowding: one note often filled several of the top results, so search must keep only the best piece per note.
What chunking a note changes
| Place | Today | With pieces | If not handled |
|---|---|---|---|
Launch sweep (NoteVectorCleanupWorker.sweepOrphans) |
deletes any vector whose id is not a note id | a piece id (noteId#2) is not a note id |
every piece is deleted on each launch. Must be done first |
Delete queue (purgeNotesInTxn → PendingVectorDelete) |
deletes the vector noteId |
must delete every piece | pieces of deleted notes stay searchable |
Search (TostoreVectorRepositoryImpl.search) |
vector id = note id | map pieces back to the note, keep the best piece | one note fills the top results |
| Small-model cut-off (0.72) | calibrated on whole notes and PDF chunks | piece scores may differ | re-measure with the calibration tests |
Knowledge extraction (ExtractKnowledgeUseCase) |
one vector per note to find similar notes, self-excluded by id | which vector, and exclude the note's own pieces | the note finds itself as "similar" |
| Manas scope (#289) | filters by note ids | must include piece ids | long notes vanish from a Manas chat |
Embedding worker (NoteEmbeddingWorker) |
one embed per note | split, embed each piece, remove the queue row after the last one | half-embedded notes after a kill (fine if piece ids are stable) |
Plan
- Build chunking for notes over ~1,100 characters on the 256 file, with every row above handled, and tests for: deleting a note removes all its pieces; the launch sweep keeps pieces of live notes; search returns each note once.
- Re-measure on the phone with the same set as above (plus some notes over 4,500 characters, which the 1024 file cuts today): right note first and top 3 for long-note and short-note questions, crowding, time, and the cut-off calibration.
- Keep it only if the results stay as good as whole notes on the 1024 file, short-note questions included. Then remove
gecko_1024_quant.tfliteand the switching code inEmbeddingService. - If it is not as good, keep #295 as it is and close this.
Refs #266, #295.
- 主要言語
- Dart
- スター
- 2
- フォーク
- 2
- 平均マージ
- 1日 35分
- マージ済み PR（30日）
- 36
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
basictech01/uniun のほかの issue
-
bug good first issue
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
basictech01/uniun#300 ·
メンテナーはふだん 1 日以内に返信
-
bug good first issue
難易度 3/5 1〜2日 初心者へのやさしさ 52/100
basictech01/uniun#301 ·
メンテナーはふだん 1 日以内に返信
-
bug good first issue P2-Medium
難易度 4/5 1〜2日 初心者へのやさしさ 22/100
basictech01/uniun#293 ·
メンテナーはふだん 1 日以内に返信
-
bug P1-High
難易度 4/5 3〜5日 初心者へのやさしさ 18/100
basictech01/uniun#286 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
P3-Low
難易度 4/5 1〜2日 初心者へのやさしさ 30/100
basictech01/uniun#282 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
basictech01/uniun の issue をすべて見る
似ている issue
-
good-first-issue P3
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
flutter/dart-intellij-third-party#758 ·
メンテナーはふだん 1 日以内に返信
-
Bug: Incorrect location coordinates on logged Sound Meter data対応中かも @Jashan32 が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
area/archive bug
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
DarkFlippers/qUnleashed#374 ·
メンテナーはふだん 1 日以内に返信
-
agent:in-progress agent:ready feature P2 · 中 区域: 悬浮窗
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
ASDW51/DanmuFloat#178 ·
メンテナーはふだん 1 日以内に返信
-
Missing workoutsオープンenhancement
難易度 2/5 1〜3時間 初心者へのやさしさ 67/100
InlitX/GymMane#237 · コメント 1 件 ·
メンテナーはふだん 5 日以内に返信