feat(rag): Hybrid BM25 + Semantic Vector Search in SQLite via Reciprocal Rank Fusion (RRF)

Open
  • Is something wrong?
#11 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
ollama, python, sqlite
Domain
backend, databases, search

Research direction

Start with tether/store.py and trace how search_chunks() queries chunks_fts and returns ChunkHit records. Then inspect the existing providers/ transport and tests for search and embedding behavior. Implement and verify the vector storage, embedding fallback, and RRF acceptance criteria; done means semantic matches can be retrieved while unconfigured embeddings preserve BM25 behavior.

Written by the indexing model from the issue text.

Description

enhancement

Summary & Motivation

Currently, tether/store.py uses SQLite FTS5 (chunks_fts with porter unicode61 stemmer and BM25 ranking) for mem search and agent query.

While BM25 excels at exact identifiers, error codes, and function names (ShowCreation, rate_limit, obs_3), it fails on conceptual / semantic / cross-lingual queries where the vocabulary differs from the source text (e.g., searching "authentication timeout" when the chunk says "JWT session expired after 30s", or searching in English over Portuguese observations).

However, Rule 7 (Zero Framework Bloat) explicitly forbids heavy external vector databases like chromadb, qdrant, milvus, or faiss. Do we even need an external Vector DB? No—we can implement Hybrid Lexical (BM25) + Dense Semantic Vector Search directly inside the existing .tether/tether.db SQLite file using Reciprocal Rank Fusion (RRF).

Literature & State-of-the-Art Grounding

  1. Why Hybrid (BM25 + Dense Vector) Beats Pure Vector DBs:
    • Thakur et al. (NeurIPS 2021), "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models" (arXiv:2104.08663): Proves that dense vector embeddings alone frequently underperform BM25 on exact entities, code symbols, and out-of-domain terminology, whereas Hybrid Retrieval (BM25 + Dense Embeddings) achieves state-of-the-art recall across both exact-keyword and semantic queries.
    • Karpukhin et al. (EMNLP 2020), "Dense Passage Retrieval (DPR)" (arXiv:2004.04906).
  2. Score-Agnostic Fusion via Reciprocal Rank Fusion (RRF):
    • Cormack, Clarke, & Buettcher (SIGIR 2009), "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods": Because BM25 scores are unbounded log-likelihoods and cosine similarities are bounded in $[-1, 1]$, linear score combination is brittle. Reciprocal Rank Fusion (RRF):
      $$\text{RRF_Score}(d) = \sum_{m \in {\text{BM25}, \text{Vec}}} \frac{1}{k + \text{rank}_m(d)} \quad (k = 60)$$
      requires zero score calibration and consistently outperforms individual rankers.
  3. Matryoshka / Int8 Quantized Embeddings in Single-File SQLite:
    • Kusupati et al. (NeurIPS 2022), "Matryoshka Representation Learning" (arXiv:2205.13147): Allows truncating embeddings (e.g., to 256 or 384 dimensions) with minimal accuracy loss. Stored as compact binary BLOB (struct.pack(f'{dim}f', ...) or int8 quantized) in SQLite, scanning 5,000 chunks in pure Python (math.fma / array('f')) or optional sqlite-vec takes < 3 milliseconds without any external server.

State-of-the-Art Fix & Architecture Design

1. Single-File SQLite Vector Table (chunk_vectors in tether/store.py)

Keep .tether/tether.db as the single source of truth (zero external vector DB servers):

CREATE TABLE IF NOT EXISTS chunk_vectors (
    rowid INTEGER PRIMARY KEY,  -- 1:1 with chunks_fts.rowid
    dim INTEGER NOT NULL,
    embedding BLOB NOT NULL     -- compact IEEE 754 float32 array ('f')
);
  • If the lightweight sqlite-vec extension is installed, optionally load it via conn.load_extension; otherwise, use a fast pure-Python array('f') cosine similarity scan over candidate chunks (zero mandatory dependencies).
2. Thin httpx Embedding Provider (tether/providers/)

Reuse the existing httpx transport in providers/ (TETHER_EMBED_MODEL or Harness(embed_model=...)):

  • Local Zero-Cost Default: ollama/nomic-embed-text or ollama/all-minilm via Ollama /api/embed.
  • Remote Compatible: OpenAI /v1/embeddings (text-embedding-3-small).
  • Graceful Zero-Config Fallback: If no embedding model is configured or reachable, store.search_chunks() transparently falls back to 100% FTS5 BM25 (existing behavior).
3. Hybrid Search with Reciprocal Rank Fusion (RRF, $k=60$)

When store.search_chunks(query, top_k=3) is called:

  1. Lexical Ranker: Fetch top-20 candidates from chunks_fts using BM25 (rank_bm25).
  2. Semantic Ranker: If embeddings are enabled, embed query and rank chunks by cosine similarity (rank_vec), so chunks that match zero exact keywords (pure semantic/synonym match) are still retrieved.
  3. RRF Merge: Fuse both ranked lists using $\sum \frac{1}{60 + \text{rank}}$ and return the top-$K$ ChunkHit records.
4. Acceptance Criteria
  • Zero heavy dependencies added (no chromadb, faiss, or numpy required; uses stdlib array/struct + httpx + optional sqlite-vec).
  • Retrieves relevant chunks even when query and document share zero overlapping words (pure semantic match test).
  • Preserves sub-millisecond pure-BM25 fallback when embed_model=None (100% backward-compatible with v0.0.3).
Dominant language
Python
Stars
3
Forks
1
Avg merge
52m
Merged PRs (30d)
166

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from igormcsouza/tether

All issues in igormcsouza/tether

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.