Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

EmbeddingService: use flutter_gemma batch API (generateEmbeddings) for sync-burst throughput

Aperta
  • Qualcosa non va?
#125 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

enhancement P0-Critical

Problem

EmbeddingService.embed (lib/features/shiv/rag/embedding/embedding_service.dart) calls EmbeddingModel.generateEmbedding(text) once per note. Every call
pays the full LiteRT per-invocation overhead (input/output tensor marshalling, GPU dispatch setup/teardown). The actual model math on Gecko-110M is
tiny — overhead dominates.

This is invisible when a user adds one note at a time, but becomes visible during sync bursts:

  • First app open after install / restore (relay backfills 100s of notes)
  • Returning online after a few days
  • Joining / catching up on a busy channel or long thread

In those windows the embedder lane saturates EmbeddingQueue(Semaphore(2)) for ~tens of seconds, RAG context lags behind the visible feed (Shiv answers
are "incomplete" until the index catches up), and the phone stays warm.

Proposal

Wire flutter_gemma's batch entry point through the service:

Future<List<List>> embedBatch(
List texts, {
bool isDocument = false,
});

Internally calls _model!.generateEmbeddings(texts, taskType: ...) — one native dispatch for the whole list — then L2-normalises each vector. Keep the
existing single-text embed() as a thin wrapper around embedBatch([text]).first so call sites that genuinely embed one string at a time don't change.

Call sites to migrate

  • The note-ingest path that embeds inbound Kind 1 events as the Gateway writes them to Isar. Currently iterates and calls embed(...) per note; should
    accumulate a small list (e.g. up to N=16 or whatever flushes a Gateway batch) and call embedBatch.
  • Saved-note / reference-note backfill jobs.

Out of scope — explicitly NOT changing

  • The embedding model itself (stay on bundled Gecko-1024 — offline-first, ~25 MB).
  • The 1-hop GraphRAG expansion. Edges come from e/t tags, not from the embedder.
  • The EmbeddingQueue(Semaphore(2)) parallelism cap — that stays as a backpressure guard.
  • Shiv chat query embedding — already single-string, no batching opportunity.

Expected user-visible improvements

Image

No change to single-add latency or retrieval quality.

Acceptance

  • New embedBatch(List) lands on EmbeddingService with GPU→CPU fallback preserved.
  • Inbound note handler in the Gateway batches embed calls.
  • Device test (one app install, integration_test/) ingests N≈100 notes and asserts wall-clock time drops by ≥5× vs. current per-note path on the same
    device.
  • No regression on chat first-token latency or single-note add.

References

Lingua principale
Dart
Stelle
2
Fork
2
Merge medio
1g 7h
PR unite (30g)
35

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di basictech01/uniun

Tutte le issue di basictech01/uniun

Issue simili

Altre issue su Dart

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.