Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Containment recall scans a base's entities for every new name, and holds the database at its CPU limit on prose-heavy sources

Open
  • Is something wrong?
#1,322 6 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
postgresql, rust
Domain
databases, performance

Research direction

Start with crates/utopia-store/src/resolution.rs, especially containment_reviews and the two LIKE queries cited in the issue; read #1104 and #1320 for the related boundary and escaping work. First benchmark both query directions on a seeded base, as suggested in #1318. Done means agreeing on whether this belongs with #1104 and whether pg_trgm is suitable, then measuring an approach that preserves the intended containment matches without the reported scans.

Written by the indexing model from the issue text.

Description

enhancement

Hi! First, thank you for how quickly and thoughtfully you've worked through #1293 and #1296. Merging #1307–#1311 made a big difference on our side. There's one more cost on the resolution path I'd love your view on before writing any code, because it runs right into the word-boundary work you have planned in #1104.

What we see

We now push Lore's agent memories and episodes into Utopia as ordinary api documents: prose, extracted like any upload. Each new named entity they mention runs containment_reviews, and that query has become the slowest thing in our database:

  • Median 12 s per call, up to 99 s, about ten calls a minute.
  • During a backfill of existing memories, 25 copies at once at roughly 1.5 minutes each. That held the database at its 4-CPU limit, and 207 extraction jobs timed out waiting for a connection in five minutes. We paused the backfill.

As far as we can tell the reason is simple: both directions are LIKE '%…%' (resolution.rs:795-796 on entities, 813-814 on name facts), so no index can help, and every new name reads the base's entities and name facts.

For prose this recall is worth keeping. "Holmes" and "Sherlock Holmes" really should meet in review. So we don't want to skip it, only to make it cheap.

A shape that might fit #1104

Once #1104 gives Latin-script containment a word boundary, each direction seems to have an indexable form:

  1. "The new name occurs inside an existing name" (lower(canonical_name) LIKE '%' || $3 || '%'): a trigram GIN index (pg_trgm, gin_trgm_ops) on lower(canonical_name) and on the name facts' lower(object_value->>'value') lets Postgres serve this LIKE from the index. The word-boundary check from #1104 would then run on the few rows it returns.
  2. "An existing name occurs inside the new name" ($3 LIKE '%' || lower(canonical_name) || '%'): no index serves this direction as written. With a word boundary, though, the contained name has to be a run of whole words of the new name. So we could enumerate those runs (a six-word name has 21) and look them up exactly with lower(canonical_name) = ANY($runs), which the existing name lookups already index.

The results should be the same pairs as today's containment with #1104's boundary, just without the scans. Scripts without word separators (CJK) would keep today's path for direction 2.

Questions before any code

  • Does this fit how you're planning #1104, or would you rather it land as part of that work?
  • Is adding the pg_trgm extension fine for Utopia's deployments, or would you prefer to avoid a new extension?
  • Would you like a bench first? I'm happy to measure both directions on a seeded base, the way #1318's description does, before opening anything.

If it helps, I'd keep it to one small PR per direction, after #1320 (the LIKE escape) is in, so the two don't touch the same lines at once.

Thank you again. It's been a real pleasure working on this with you.

Related: #1104, #1296, #1293, #1320

Dominant language
Rust
Stars
8.1k
Forks
1.1k
Avg merge
21h 30m
Merged PRs (30d)
251

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deeplethe/utopia

All issues in deeplethe/utopia

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.