Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Memory and SQLite term indexes normalize terms differently

Cerrado
  • ¿Algo no está bien?
#322 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@bmerkle ya está trabajando en esto.

Desde el 28/9/2026.

  • #325 de @bmerkle — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
56/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python, sqlite
Área
backend, databases

Línea de trabajo

Start by comparing _prepare_term in storage/memory/semrefindex.py and storage/sqlite/semrefindex.py, then inspect make_property_term_text and the corresponding PropertyIndex implementations in storage/memory/propindex.py and storage/sqlite/propindex.py. Run the existing memory and SQLite index tests around add, lookup_term, get_terms, and size. Done means both providers follow one deliberate normalization contract, including the PropertyIndex behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Severity: High — the same query hits in one backend and misses in the other
Area: storage/memory/semrefindex.py, storage/sqlite/semrefindex.py

Summary

Both TermToSemanticRefIndex implementations funnel terms through a
_prepare_term hook before storing and before looking up, but the two hooks do
different things. Memory only lowercases; SQLite also strips, NFC-normalizes and
collapses whitespace. Terms that the SQLite index treats as equal are distinct
in memory, so lookups, get_terms() and size() all disagree.

Reproduction

await idx.add_term("Space  Needle ", 1)
print(sorted(await idx.get_terms()))
print(await idx.lookup_term("space needle"))

memory  get_terms()=['space  needle ']   lookup_term('space needle') -> []
sqlite  get_terms()=['space needle']     lookup_term('space needle') -> [ScoredSemanticRefOrdinal(1, 1.0)]

Further divergences:

property input memory sqlite
lookup_term add ('A', 0), probe 'A ' [] [(0, 1.0)]
size() add (' ', 0), ('\t', 0) 2 1
get_terms() add (' ', 0) [' '] ['']

Root cause

storage/memory/semrefindex.py:741-742:

def _prepare_term(self, term: str) -> str:
    return term.lower()

storage/sqlite/semrefindex.py:172-184:

def _prepare_term(self, term: str) -> str:
    term = term.strip()
    term = unicodedata.normalize("NFC", term)
    term = re.sub(r"\s+", " ", term)
    return term.lower()

Each backend is internally consistent — it normalizes the same way on write and
read — so neither is broken on its own. The defect is that normalization is a
property of the index contract, not of the storage engine, and it has been
implemented twice.

Impact

  • Recall differs between deployments using different providers.
  • A conversation indexed in memory and serialized into SQLite (or vice versa)
    will not match the terms it was built with, because the keys were normalized
    under different rules.
  • Whitespace-only and empty terms are conflated in SQLite but not in memory,
    which also shifts size().

Suggested fix

Hoist normalization into one shared function (alongside make_property_term_text
in a neutral module, or on the protocol itself) and have both implementations
call it. The SQLite version looks like the intended behavior; promoting it to the
shared definition is probably the right direction, but that changes existing
memory-backed index keys, so it needs a deliberate call.

PropertyIndex._prepare_term has the same shape of duplication
(storage/memory/propindex.py:316 vs the inline .lower() at
storage/sqlite/propindex.py:55) and should move with it.

Lenguaje dominante
Python
Estrellas
892
Forks
76
Merge medio
2 d 15 h
PR fusionados (30 d)
23

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/typeagent-py

Todos los issues de microsoft/typeagent-py

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.