MemoryRetriever: rare query words weigh the same as common ones; on 20 real prompts a floor alone did not help, IDF in the score with the floor did (12 → 20 useful notes)
Los mantenedores suelen responder en 5 días
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Aptitud para principiantes
- 42/100
- Tipo de issue
- Nueva funcionalidad
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Stack tecnológico
- typescript
- Área
- search
Línea de trabajo
Start with scoreNote and computeRelevantContext in LIFEOS/TOOLS/MemoryRetriever.ts, which the issue says give every query term equal weight. The proposed weightedByRarity sketch in the body shows where IDF weights and a matched-information floor would go, and the issue also names the getRelevantContext call used by MemoryTurnStart. Done means rare terms pass on their own, a prompt of only common words no longer brings in the topical note, and the background-set tests described in the issue pass.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Summary
In 8.761.0, scoreNote in LIFEOS/TOOLS/MemoryRetriever.ts gives every query term the same weight. A rare word the prompt is about counts no more than a common word. On long, dictated prompts, the common words win the top 5.
Two parts of #2207 are still open there: IDF and the emission floor. This issue reports a measurement on real conversational prompts that separates the two. A floor alone did not help. IDF in the score together with the floor did. A small patch follows.
Example
The prompt is about 30 words of Portuguese and names one tool. A note titled with that tool's name exists. It does not reach the top 25. The top 5 are notes whose titles share three common words with the prompt ("veja", "mesmas", "configurações", roughly "see", "same", "settings"). Each common word scores the same +10 title points as the tool name.
Measurement
- Setup. One install with a mostly Portuguese corpus of 674 notes (467 knowledge, 207 learning).
- Prompts. 20 of the principal's real prompts, drawn with a fixed seed from recent sessions, each at least 8 words.
- Retriever.
getRelevantContext(prompt, { topK: 5, threshold: 0.20, allowStaleIndex: true }), called the wayMemoryTurnStartcalls it. - Labels. Every (prompt, note) pair from every arm was pooled and judged blind by a model that did not know which arm produced it. The question was: "would this note give the assistant useful information to understand or answer this prompt?"
- Tokenizer. Every arm uses a Unicode tokenizer with plural folding; the stock one drops accented letters (see my comment on #2207). That makes the arms comparable on Portuguese. It does not change how terms are weighted.
| Scoring | Useful notes in the top 5 (of 100) | Prompts with at least one useful note |
|---|---|---|
8.761.0 scoreNote, every term weighted equally |
12 | 7 |
| + matched-information floor only | 11 | 6 |
| + floor + IDF weight in the score | 20 | 9 |
Caveats: one install, 20 prompts, one run. About half of the labels judged the note from its title and frontmatter rather than the whole body. Many prompts were replies that lean on earlier turns, which caps every arm.
Why the floor alone did not help: in a long prompt, many common words together pass any floor based on summed term weight. The ranking among the notes that pass is still unweighted.
This seems to differ from @Hoyass's result on #2207, where IDF added no hits. Their questions were short and written against the corpus. Long dictated prompts carry many common words, so that may be where IDF starts to matter.
Proposed change
Weight each term by BM25 IDF, normalized so a word present in 1% of notes (at least 2) weighs 1, capped at 1. Multiply that weight into the title, tag, related and body contributions. A note enters only if the summed weights of the query terms it contains reach 0.75. A rare word passes alone, a common word does not, and several common words with one rare word pass.
const RARE_DOC_FRACTION = 0.01;
const RARE_DOC_MIN = 2;
const MIN_MATCHED_INFORMATION = 0.75;
function weightedByRarity(notes: KnowledgeNote[], queryTerms: string[], avgDocLength: number): ScoredNote[] {
// points of each term in each note, by the existing scoreNote; > 0 means the note has the term
const perTerm = notes.map((note) => queryTerms.map((t) => scoreNote(note, [t], avgDocLength)));
const n = notes.length;
const idf = (d: number) => Math.log(1 + (n - d + 0.5) / (d + 0.5));
const dfRare = Math.min(n, Math.max(RARE_DOC_MIN, Math.ceil(RARE_DOC_FRACTION * n)));
const weights = queryTerms.map((_, j) => {
const df = perTerm.reduce((c, row) => c + (row[j]! > 0 ? 1 : 0), 0);
return df === 0 ? 0 : Math.min(1, idf(df) / idf(dfRare));
});
const scored: ScoredNote[] = [];
notes.forEach((note, i) => {
let info = 0, score = 0;
perTerm[i]!.forEach((points, j) => { if (points > 0) info += weights[j]!; score += weights[j]! * points; });
if (info >= MIN_MATCHED_INFORMATION) scored.push({ note, score });
});
return scored;
}
// in computeRelevantContext, instead of mapping scoreNote over allNotes:
let scored: ScoredNote[] = weightedByRarity(allNotes, queryTerms, avgDocLength)
.filter((s) => s.score >= threshold)
.sort((a, b) => b.score - a.score);
Cost. Over the same 20 prompts, median retrieval time went from 47 ms to 66 ms and the maximum from 115 ms to 171 ms.
Controls. Tests on a background set of 30 notes made of common words were run with the patch:
- "the cat and the dog" and "não sei se chove amanhã" ("I don't know if it rains tomorrow") no longer bring in the topical note;
- a single rare word still brings its notes in a corpus of 3,000;
- a 75-word dictated prompt still finds the note for one of its three topics.
I can open a PR with the patch and the tests if this direction is wanted. It does not settle the larger question in #2207, whether per-turn keyword retrieval is the right mechanism at all.
- Lenguaje dominante
- TypeScript
- Estrellas
- 19.9k
- Forks
- 2.5k
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de danielmiessler/LifeOS
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 67/100
danielmiessler/LifeOS#2894 ·
Los mantenedores suelen responder en 5 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
danielmiessler/LifeOS#2891 ·
Los mantenedores suelen responder en 5 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
danielmiessler/LifeOS#2888 ·
Los mantenedores suelen responder en 5 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
danielmiessler/LifeOS#2886 ·
Los mantenedores suelen responder en 5 días
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 68/100
danielmiessler/LifeOS#2848 ·
Los mantenedores suelen responder en 5 días
Todos los issues de danielmiessler/LifeOS
Issues similares
-
[功能请求] 设置选项语言调整Abierto功能请求
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 84/100
antithesishq/bombadil#473 ·
Los mantenedores suelen responder en 2 días
-
The structured output reference promises null results for refusals that carry validation detailsAbiertobug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
mattermost/mattermost-mobile#10358 ·
Los mantenedores suelen responder en 1 día
-
ready-for-agent
Dificultad 1/5 Menos de una hora Aptitud para principiantes 86/100
LucasSantana-Dev/Lucky#3257 ·
Los mantenedores suelen responder en 1 día