FTS search misses accented words: trigram tokenizer without remove_diacritics

Closed Beginner friendly
  • Is something wrong?
#2,210 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@dnlrsls is already working on this.

Since Nov 1, 2026.

  • #2219 by @dnlrsls — merged

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
85/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, sqlite
Domain
search

Research direction

Find where the two FTS5 tables are created (search the codebase for tokenize='trigram' in the schema definitions for observations_fts and prompts_fts) and add remove_diacritics 1. Then add an index-rebuild path for existing stores, e.g. the INSERT INTO observations_fts(observations_fts) VALUES('rebuild') command the issue suggests. Verify with the SQL repro from the issue and any existing search tests; done means accented and unaccented queries both match.

Written by the indexing model from the issue text.

Description

status:needs-review

Problem

mem_search does not fold accents. In languages that use diacritics (Spanish, French, Portuguese…), a query without the accent misses memories that contain it, and the other way round. People often type without accents, so these misses are frequent.

Example from a real store (engram 3.6.0, SQLite 3.50.4): a memory titled «resumen del vídeo de Fazt» is not found by mem_search "video".

Cause

Both FTS tables are created with tokenize='trigram':

CREATE VIRTUAL TABLE observations_fts USING fts5(title, content, tool_name, type, project, topic_key,
  tokenize='trigram', content='observations', content_rowid='id');
CREATE VIRTUAL TABLE prompts_fts USING fts5(content, project,
  tokenize='trigram', content='user_prompts', content_rowid='id');

Minimal reproduction (SQLite 3.50.4)

CREATE VIRTUAL TABLE t USING fts5(x, tokenize='trigram');
INSERT INTO t VALUES ('resumen del vídeo de Fazt');
SELECT count(*) FROM t WHERE t MATCH 'video';   -- 0

CREATE VIRTUAL TABLE t2 USING fts5(x, tokenize='trigram remove_diacritics 1');
INSERT INTO t2 VALUES ('resumen del vídeo de Fazt');
SELECT count(*) FROM t2 WHERE t2 MATCH 'video';  -- 1

Proposed fix

Create both tables with tokenize='trigram remove_diacritics 1' (available since SQLite 3.45), then rebuild the index for existing stores, for example with INSERT INTO observations_fts(observations_fts) VALUES('rebuild') after recreating the table. The change keeps substring matching and only folds diacritics.

Workaround today

Searching twice, once with and once without the accent.

Dominant language
Go
Stars
7.1k
Forks
729
Avg merge
10h 46m
Merged PRs (30d)
268

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Gentleman-Programming/engram

All issues in Gentleman-Programming/engram

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.