Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Memory and SQLite term indexes normalize terms differently

已关闭
  • 哪里不对？
#322 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@bmerkle 已经在做这个了。

开始于 2026年9月28日。

  • #325 来自 @bmerkle —— 未关闭

评估

难度
4/5
预计耗时
3-5 天
新手友好度
56/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
python, sqlite
领域
backend, databases

调研方向

Start by comparing _prepare_term in storage/memory/semrefindex.py and storage/sqlite/semrefindex.py, then inspect make_property_term_text and the corresponding PropertyIndex implementations in storage/memory/propindex.py and storage/sqlite/propindex.py. Run the existing memory and SQLite index tests around add, lookup_term, get_terms, and size. Done means both providers follow one deliberate normalization contract, including the PropertyIndex behavior.

由索引模型根据 Issue 内容生成。

描述

Severity: High — the same query hits in one backend and misses in the other
Area: storage/memory/semrefindex.py, storage/sqlite/semrefindex.py

Summary

Both TermToSemanticRefIndex implementations funnel terms through a
_prepare_term hook before storing and before looking up, but the two hooks do
different things. Memory only lowercases; SQLite also strips, NFC-normalizes and
collapses whitespace. Terms that the SQLite index treats as equal are distinct
in memory, so lookups, get_terms() and size() all disagree.

Reproduction

await idx.add_term("Space  Needle ", 1)
print(sorted(await idx.get_terms()))
print(await idx.lookup_term("space needle"))

memory  get_terms()=['space  needle ']   lookup_term('space needle') -> []
sqlite  get_terms()=['space needle']     lookup_term('space needle') -> [ScoredSemanticRefOrdinal(1, 1.0)]

Further divergences:

property input memory sqlite
lookup_term add ('A', 0), probe 'A ' [] [(0, 1.0)]
size() add (' ', 0), ('\t', 0) 2 1
get_terms() add (' ', 0) [' '] ['']

Root cause

storage/memory/semrefindex.py:741-742:

def _prepare_term(self, term: str) -> str:
    return term.lower()

storage/sqlite/semrefindex.py:172-184:

def _prepare_term(self, term: str) -> str:
    term = term.strip()
    term = unicodedata.normalize("NFC", term)
    term = re.sub(r"\s+", " ", term)
    return term.lower()

Each backend is internally consistent — it normalizes the same way on write and
read — so neither is broken on its own. The defect is that normalization is a
property of the index contract, not of the storage engine, and it has been
implemented twice.

Impact

  • Recall differs between deployments using different providers.
  • A conversation indexed in memory and serialized into SQLite (or vice versa)
    will not match the terms it was built with, because the keys were normalized
    under different rules.
  • Whitespace-only and empty terms are conflated in SQLite but not in memory,
    which also shifts size().

Suggested fix

Hoist normalization into one shared function (alongside make_property_term_text
in a neutral module, or on the protocol itself) and have both implementations
call it. The SQLite version looks like the intended behavior; promoting it to the
shared definition is probably the right direction, but that changes existing
memory-backed index keys, so it needs a deliberate call.

PropertyIndex._prepare_term has the same shape of duplication
(storage/memory/propindex.py:316 vs the inline .lower() at
storage/sqlite/propindex.py:55) and should move with it.

主要语言
Python
星标
892
派生
76
平均合并
2 天 15 小时
30 天内合并 PR
23

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

microsoft/typeagent-py 的其他 Issue

查看 microsoft/typeagent-py 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。