retrieve misses Greek words with ΐ or ΰ when case differs

Abierto Apto para principiantes
#337 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
77/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, typescript
Área
backend, search

Línea de trabajo

Lee _fold en src/selfgraph/retrieval.py y fold en typescript/src/retrieval.ts; después, revisa las pruebas de retrieval y la descripción de la normalización en docs/retrieval.md. Reproduce en ambas implementaciones la discrepancia griega entre mayúsculas y minúsculas y confirma que la corrección hace que la consulta y el texto almacenado coincidan. Se considera terminado cuando funcionan los ejemplos griegos indicados y se mantienen las coincidencias existentes con diferencias de mayúsculas y minúsculas en caracteres latinos.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

approved-for-pr area: retrieval bug effort: S P3

retrieve is documented as matching without regard to case, but it misses Greek words that contain ΐ or ΰ (a letter with both dialytika and tonos) when the query and the entity differ in case. An entity named "Πρωτεΐνη" ("protein") is not seeded by "ΠΡΩΤΕΪ́ΝΗ", which is what "Πρωτεΐνη".upper() returns, and the reverse fails as well. Latin cases such as Paris/PARIS, Zoë/ZOË, and Straße/STRASSE work.

Tested at 5a277c9 (current main), Python 3.14, Kuzu 0.11.3, on macOS. The cause is in the token fold, not the store.

Reproduce
import asyncio, os, tempfile
from selfgraph import KuzuStore, SelfGraph

async def main():
    async with SelfGraph(store=KuzuStore(os.path.join(tempfile.mkdtemp(), "g.kuzu"))) as g:
        await g.entities.create("Πρωτεΐνη", "Concept", id="concept:protein")
        await g.entities.create("Paris", "Place", id="place:paris")
        for q in ["Πρωτεΐνη", "Πρωτεΐνη".upper(), "πρωτεΐνη", "PARIS"]:
            print(repr(q), [s.id for s in (await g.retrieve(q)).seeds])

asyncio.run(main())
'Πρωτεΐνη' ['concept:protein']
'ΠΡΩΤΕΪ́ΝΗ' []
'πρωτεΐνη' ['concept:protein']
'PARIS' ['place:paris']

Storing the entity as "ΠΡΩΤΕΪ́ΝΗ" and querying "πρωτεΐνη" also returns no seeds. The same happens with "Ταΐζω" and "ΤΑΪ́ΖΩ", and with any other word that has ΐ (U+0390) or ΰ (U+03B0).

Cause

_fold (src/selfgraph/retrieval.py:161) is normalize_text(text).casefold(): NFC first, then casefold. Casefolding does not keep text in NFC:

import unicodedata as u
lower, upper = "Πρωτεΐνη", "Πρωτεΐνη".upper()
fold = lambda t: u.normalize("NFC", t).casefold()
print(fold(lower) == fold(upper))                                     # False
print(u.normalize("NFC", fold(lower)) == u.normalize("NFC", fold(upper)))  # True

The lowercase ΐ (U+0390) folds to U+03B9 U+0308 U+0301. The uppercase form's NFC is Ϊ plus a combining tonos (U+03AA U+0301), which folds to U+03CA U+0301. The two are canonically equivalent but not byte-equal, so neither the BM25 terms nor the substring boost match.

docs/retrieval.md ("How it works", step 1) says the query is NFC-normalized and casefolded so that query and stored text compare in one form. Unicode's canonical caseless match needs one more normalization after the fold, NFC(casefold(NFC(x))) or the NFD equivalent.

The TypeScript fold (typescript/src/retrieval.ts:87-88) is casefold(normalizeText(text)), using the full case-folding table in typescript/src/internal/casefold.ts, so it gives the same tokens and misses the same matches.

Related: #312 has the same root (case mapping applied after normalization leaves non-NFC text), in entity_id.

Lenguaje dominante
TypeScript
Estrellas
173
Forks
58
Merge medio
6 h
PR fusionados (30 d)
120

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de hydra-db/selfgraph

Todos los issues de hydra-db/selfgraph

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.