MemoryRetriever: rare query words weigh the same as common ones; on 20 real prompts a floor alone did not help, IDF in the score with the floor did (12 → 20 useful notes)
Maintainers usually reply within 5 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 42/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- typescript
- Domain
- search
Research direction
Start with scoreNote and computeRelevantContext in LIFEOS/TOOLS/MemoryRetriever.ts, which the issue says give every query term equal weight. The proposed weightedByRarity sketch in the body shows where IDF weights and a matched-information floor would go, and the issue also names the getRelevantContext call used by MemoryTurnStart. Done means rare terms pass on their own, a prompt of only common words no longer brings in the topical note, and the background-set tests described in the issue pass.
Written by the indexing model from the issue text.
Description
Summary
In 8.761.0, scoreNote in LIFEOS/TOOLS/MemoryRetriever.ts gives every query term the same weight. A rare word the prompt is about counts no more than a common word. On long, dictated prompts, the common words win the top 5.
Two parts of #2207 are still open there: IDF and the emission floor. This issue reports a measurement on real conversational prompts that separates the two. A floor alone did not help. IDF in the score together with the floor did. A small patch follows.
Example
The prompt is about 30 words of Portuguese and names one tool. A note titled with that tool's name exists. It does not reach the top 25. The top 5 are notes whose titles share three common words with the prompt ("veja", "mesmas", "configurações", roughly "see", "same", "settings"). Each common word scores the same +10 title points as the tool name.
Measurement
- Setup. One install with a mostly Portuguese corpus of 674 notes (467 knowledge, 207 learning).
- Prompts. 20 of the principal's real prompts, drawn with a fixed seed from recent sessions, each at least 8 words.
- Retriever.
getRelevantContext(prompt, { topK: 5, threshold: 0.20, allowStaleIndex: true }), called the wayMemoryTurnStartcalls it. - Labels. Every (prompt, note) pair from every arm was pooled and judged blind by a model that did not know which arm produced it. The question was: "would this note give the assistant useful information to understand or answer this prompt?"
- Tokenizer. Every arm uses a Unicode tokenizer with plural folding; the stock one drops accented letters (see my comment on #2207). That makes the arms comparable on Portuguese. It does not change how terms are weighted.
| Scoring | Useful notes in the top 5 (of 100) | Prompts with at least one useful note |
|---|---|---|
8.761.0 scoreNote, every term weighted equally |
12 | 7 |
| + matched-information floor only | 11 | 6 |
| + floor + IDF weight in the score | 20 | 9 |
Caveats: one install, 20 prompts, one run. About half of the labels judged the note from its title and frontmatter rather than the whole body. Many prompts were replies that lean on earlier turns, which caps every arm.
Why the floor alone did not help: in a long prompt, many common words together pass any floor based on summed term weight. The ranking among the notes that pass is still unweighted.
This seems to differ from @Hoyass's result on #2207, where IDF added no hits. Their questions were short and written against the corpus. Long dictated prompts carry many common words, so that may be where IDF starts to matter.
Proposed change
Weight each term by BM25 IDF, normalized so a word present in 1% of notes (at least 2) weighs 1, capped at 1. Multiply that weight into the title, tag, related and body contributions. A note enters only if the summed weights of the query terms it contains reach 0.75. A rare word passes alone, a common word does not, and several common words with one rare word pass.
const RARE_DOC_FRACTION = 0.01;
const RARE_DOC_MIN = 2;
const MIN_MATCHED_INFORMATION = 0.75;
function weightedByRarity(notes: KnowledgeNote[], queryTerms: string[], avgDocLength: number): ScoredNote[] {
// points of each term in each note, by the existing scoreNote; > 0 means the note has the term
const perTerm = notes.map((note) => queryTerms.map((t) => scoreNote(note, [t], avgDocLength)));
const n = notes.length;
const idf = (d: number) => Math.log(1 + (n - d + 0.5) / (d + 0.5));
const dfRare = Math.min(n, Math.max(RARE_DOC_MIN, Math.ceil(RARE_DOC_FRACTION * n)));
const weights = queryTerms.map((_, j) => {
const df = perTerm.reduce((c, row) => c + (row[j]! > 0 ? 1 : 0), 0);
return df === 0 ? 0 : Math.min(1, idf(df) / idf(dfRare));
});
const scored: ScoredNote[] = [];
notes.forEach((note, i) => {
let info = 0, score = 0;
perTerm[i]!.forEach((points, j) => { if (points > 0) info += weights[j]!; score += weights[j]! * points; });
if (info >= MIN_MATCHED_INFORMATION) scored.push({ note, score });
});
return scored;
}
// in computeRelevantContext, instead of mapping scoreNote over allNotes:
let scored: ScoredNote[] = weightedByRarity(allNotes, queryTerms, avgDocLength)
.filter((s) => s.score >= threshold)
.sort((a, b) => b.score - a.score);
Cost. Over the same 20 prompts, median retrieval time went from 47 ms to 66 ms and the maximum from 115 ms to 171 ms.
Controls. Tests on a background set of 30 notes made of common words were run with the patch:
- "the cat and the dog" and "não sei se chove amanhã" ("I don't know if it rains tomorrow") no longer bring in the topical note;
- a single rare word still brings its notes in a corpus of 3,000;
- a 75-word dictated prompt still finds the note for one of its three topics.
I can open a PR with the patch and the tests if this direction is wanted. It does not settle the larger question in #2207, whether per-turn keyword retrieval is the right mechanism at all.
- Dominant language
- TypeScript
- Stars
- 19.9k
- Forks
- 2.5k
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from danielmiessler/LifeOS
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
danielmiessler/LifeOS#2894 ·
Maintainers usually reply within 5 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
danielmiessler/LifeOS#2891 ·
Maintainers usually reply within 5 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
danielmiessler/LifeOS#2888 ·
Maintainers usually reply within 5 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
danielmiessler/LifeOS#2886 ·
Maintainers usually reply within 5 days
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
danielmiessler/LifeOS#2848 ·
Maintainers usually reply within 5 days
All issues in danielmiessler/LifeOS
Similar issues
-
ready-for-agent
Difficulty 1/5 Under an hour Newbie friendliness 86/100
LucasSantana-Dev/Lucky#3257 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
XiaomiMiMo/MiMo-Code#2849 · 1 comment ·
Maintainers usually reply within 1 day
-
enhancement frontend good first issue help wanted testing-infrastructure
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 2 days
-
enhancement follow-up
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
obsidian-tasks-group/obsidian-tasks#4155 ·
Maintainers usually reply within 1 day