Drop the 1024 Gecko file: chunk long notes on the 256 file, measure first
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 25/100
调研方向
Start with docs/SHIVA/embedding.md and the named entry points: NoteVectorCleanupWorker.sweepOrphans, purgeNotesInTxn, TostoreVectorRepositoryImpl.search, ExtractKnowledgeUseCase, NoteEmbeddingWorker, and EmbeddingService. Implement stable note piece IDs and handle cleanup, deletion, search deduplication, extraction, Manas scope, and interrupted embedding; add the three planned tests. Re-measure on the phone against the listed whole-note results, and keep the change only if long-note and short-note search results stay as good.
由索引模型根据 Issue 内容生成。
描述
Goal
Ship only the 256-token Gecko file and drop the 1024-token one: the app gets 145 MB smaller (31 MB smaller than before #295), and every embed takes ~2.2 s instead of up to 11.7 s on a weak phone.
The blocker is long notes. The 256 file reads about 1,100 characters, so a longer note would be cut and its later text would not be searchable. To drop the 1024 file, notes over ~1,100 characters must be chunked: several vectors per note, each piece embedded with the 256 file.
Where things stand (#295)
- The app bundles both files. It embeds with 256 and switches to 1024 only for text that does not fit (~1.1 s a switch).
- Both files give the same vector for the same text (cosine 1.0000), so old and new vectors share one index. Vectors made by the 1024 file stay valid after it is removed.
- PDF/DOCX chunks (at most 700 characters) already fit the 256 file.
What was measured (vivo 1933, see docs/SHIVA/embedding.md)
14 long notes (950-2,000 characters), 16 short notes in the same index, and 44 questions (28 aimed at the start or the end of a long note):
| Long-note questions: first / top 3 | Short-note questions: first / top 3 | One note fills 2+ of the top 3 | Time to index | |
|---|---|---|---|---|
| Whole note, 1024 file | 25/28 · 28/28 | 13/16 · 16/16 | 0/44 | 352 s |
| 700-character chunks, 256 file | 27/28 · 28/28 | 12/16 · 15/16 | 32/44 | 109 s |
- Better: questions about a long note found it first more often (27/28 against 25/28).
- Worse: one short note dropped out of the top 3, because small pieces of long notes compete harder with short notes.
- Crowding: one note often filled several of the top results, so search must keep only the best piece per note.
What chunking a note changes
| Place | Today | With pieces | If not handled |
|---|---|---|---|
Launch sweep (NoteVectorCleanupWorker.sweepOrphans) |
deletes any vector whose id is not a note id | a piece id (noteId#2) is not a note id |
every piece is deleted on each launch. Must be done first |
Delete queue (purgeNotesInTxn → PendingVectorDelete) |
deletes the vector noteId |
must delete every piece | pieces of deleted notes stay searchable |
Search (TostoreVectorRepositoryImpl.search) |
vector id = note id | map pieces back to the note, keep the best piece | one note fills the top results |
| Small-model cut-off (0.72) | calibrated on whole notes and PDF chunks | piece scores may differ | re-measure with the calibration tests |
Knowledge extraction (ExtractKnowledgeUseCase) |
one vector per note to find similar notes, self-excluded by id | which vector, and exclude the note's own pieces | the note finds itself as "similar" |
| Manas scope (#289) | filters by note ids | must include piece ids | long notes vanish from a Manas chat |
Embedding worker (NoteEmbeddingWorker) |
one embed per note | split, embed each piece, remove the queue row after the last one | half-embedded notes after a kill (fine if piece ids are stable) |
Plan
- Build chunking for notes over ~1,100 characters on the 256 file, with every row above handled, and tests for: deleting a note removes all its pieces; the launch sweep keeps pieces of live notes; search returns each note once.
- Re-measure on the phone with the same set as above (plus some notes over 4,500 characters, which the 1024 file cuts today): right note first and top 3 for long-note and short-note questions, crowding, time, and the cut-off calibration.
- Keep it only if the results stay as good as whole notes on the 1024 file, short-note questions included. Then remove
gecko_1024_quant.tfliteand the switching code inEmbeddingService. - If it is not as good, keep #295 as it is and close this.
Refs #266, #295.
- 主要语言
- Dart
- 星标
- 2
- 派生
- 2
- 平均合并
- 19 小时 21 分钟
- 30 天内合并 PR
- 37
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
basictech01/uniun 的其他 Issue
-
bug good first issue P2-Medium
难度 2/5 1-3 小时 新手友好度 64/100
basictech01/uniun#305 ·
维护者通常 1 天内回复
-
bug good first issue
难度 2/5 1-3 小时 新手友好度 82/100
basictech01/uniun#300 ·
维护者通常 1 天内回复
-
enhancement
难度 4/5 1-2 天 新手友好度 42/100
basictech01/uniun#306 · 1 条评论 ·
维护者通常 1 天内回复
-
bug good first issue P2-Medium
难度 3/5 1-2 天 新手友好度 52/100
basictech01/uniun#304 ·
维护者通常 1 天内回复
-
bug good first issue
难度 3/5 1-2 天 新手友好度 52/100
basictech01/uniun#301 ·
维护者通常 1 天内回复
查看 basictech01/uniun 的全部 Issue
相似的 Issue
-
Standalone dash flashes as its own RSVP step (e.g. Russian dialogue)可能已有人在做 @bezig01 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
daniel-pimenta-cc/ledor#53 ·
-
enhancement firebase
难度 2/5 1-3 小时 新手友好度 65/100
brysonddelozier/scribbles#24 ·
-
难度 2/5 1-3 小时 新手友好度 70/100
daohoangson/flutter_widget_from_html#1657 ·
维护者通常 3 天内回复
-
难度 1/5 1 小时以内 新手友好度 88/100
RevenueCat/purchases-flutter#1973 ·
维护者通常 1 天内回复
-
Needs Attention platform: android type: infrastructure umbrella
难度 2/5 1-3 小时 新手友好度 78/100
firebase/flutterfire#18832 ·
维护者通常 1 天内回复