Python: Bug: Redis search returns None for every field that declares a storage_name, because RETURN lists the declared name

Abierto Apto para principiantes
  • ¿Algo no está bien?
#14,753 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 5 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
78/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, redis
Área
backend, databases

Línea de trabajo

Empieza en python/semantic_kernel/connectors/redis.py y lee _add_return_fields en ambas clases de colección y RedisHashsetCollection._deserialize_store_models_to_dicts. Después, inspecciona tests/unit/connectors/memory/test_redis_store.py y ejecuta sus pruebas. Se considera terminado cuando ambas colecciones devuelvan los nombres de almacenamiento en los resultados de búsqueda y la deserialización de vectores de Hashset produzca una lista de números de coma flotante tanto en la búsqueda como en la obtención.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

python triage
Description

When a field of a Redis collection declares a storage_name, a search returns the record without that field.

The collection creates the index attributes under storage_name or name, and it stores the values under storage_name or name. The KNN part of the search query also uses the storage name. But the RETURN clause of the query lists the declared field name. The server has no attribute with that name, so it leaves the field out.

  • A data field with a storage_name comes back as None (dict or dataclass record types). For a pydantic record type, the validation fails because the field is missing.
  • With include_vectors=True, a vector field with a storage_name is not returned either.
  • RedisHashsetCollection has a second problem in the same area. It decodes the vector bytes and writes the array under the declared name. The data model step then reads the storage name key, so the vector stays raw bytes. This also hits get, not only search.

Both RedisHashsetCollection and RedisJsonCollection have the wrong RETURN list.

Reproduction

I ran this on current main (df1cb33d0), Python 3.13, redis-py and redisvl 0.27.2. I have no Redis Stack server here. The fake ft(...).search below answers the way RediSearch does for RETURN: only the attributes named in RETURN come back, plus the key. The RETURN list it prints is the one the connector sends.

import asyncio
import struct
import warnings
from types import SimpleNamespace

warnings.filterwarnings("ignore")
from redis.asyncio import Redis
from redis.commands.search.document import Document
from redisvl.schema import IndexSchema

import semantic_kernel.connectors.redis as redis_module
from semantic_kernel.connectors.redis import RedisHashsetCollection
from semantic_kernel.data.vector import VectorStoreCollectionDefinition, VectorStoreField

DEFN = VectorStoreCollectionDefinition(
    fields=[
        VectorStoreField("key", name="id", type="str"),
        VectorStoreField("data", name="title", type="str", storage_name="doc_title", is_indexed=True),
        VectorStoreField("vector", name="vec", dimensions=3, storage_name="doc_vec", type="float"),
    ]
)
VEC = struct.pack("<3f", 0.1, 0.2, 0.3)
STORED_HASH = {"doc_title": "hello", "doc_vec": VEC, "vector_distance": "0.25"}


class FakeFT:
    async def search(self, query, query_params=None):
        args = query.get_args()
        i = args.index("RETURN")
        wanted = args[i + 2 : i + 2 + args[i + 1]]
        print("RETURN sent to the server:", wanted)
        doc = {"id": "c:k1", "payload": None}
        for name in wanted:
            if name in STORED_HASH:
                doc[name] = STORED_HASH[name]
        return SimpleNamespace(docs=[Document(**doc)], total=1)


SCHEMA = IndexSchema.from_dict(
    {
        "index": {"name": "c", "prefix": "c", "storage_type": "hash"},
        "fields": [
            {"name": "doc_title", "type": "text"},
            {
                "name": "doc_vec",
                "type": "vector",
                "attrs": {"dims": 3, "algorithm": "flat", "distance_metric": "cosine", "datatype": "float32"},
            },
        ],
    }
)


async def fake_from_existing(*args, **kwargs):
    return SimpleNamespace(schema=SCHEMA)


redis_module.AsyncSearchIndex.from_existing = fake_from_existing


async def main():
    db = Redis()
    db.ft = lambda name: FakeFT()
    col = RedisHashsetCollection(record_type=dict, definition=DEFN, collection_name="c", redis_database=db)
    print("stored attributes:", ["doc_title", "doc_vec"])
    res = await col.search(vector=[0.1, 0.2, 0.3], top=3, include_vectors=True)
    async for r in res.results:
        print("search record:", r.record)

    # the get path: a stored hash read back through the same deserialization
    store_record = {"id": "k1", "doc_title": "hello", "doc_vec": VEC}
    print("get record:", col.deserialize([store_record], include_vectors=True))


asyncio.run(main())

Output on main:

stored attributes: ['doc_title', 'doc_vec']
RETURN sent to the server: ['vector_distance', 'id', 'title', 'vec']
search record: {'id': 'c:k1', 'title': None, 'vec': None}
get record: [{'id': 'k1', 'title': 'hello', 'vec': b'\xcd\xcc\xcc=\xcd\xccL>\x9a\x99\x99>'}]

The stored hash has doc_title and doc_vec, but the query asks for title and vec. The search result has None for both fields. The get record has the title, but the vector is still bytes.

For RedisJsonCollection the generated RETURN list is the same, ['vector_distance', 'id', 'title', 'vec'], while the index attributes are doc_title and doc_vec. I printed it with col._construct_vector_query([0.1, 0.2, 0.3], VectorSearchOptions(include_vectors=True)) and query.get_args().

Cause

In python/semantic_kernel/connectors/redis.py, _add_return_fields of both collections calls query.return_field(field.name). Everything else in the file uses field.storage_name or field.name: _definition_to_redis_fields, the serializers, the KNN vector_field_name and the filter parser.

In RedisHashsetCollection._deserialize_store_models_to_dicts, the vector is decoded into rec[field.name]. VectorStoreCollection._deserialize_dict_to_data_model reads record.get(field.storage_name or field.name) for dict and dataclass types, and for pydantic types it moves the storage key over the declared name. In both cases the raw bytes win.

The same thing was reported and fixed for Azure AI Search hybrid search in #14684.

Expected

A field with a storage_name is returned by search and by get, with the same value as a field without one. The vector is decoded to a list of floats.

Possible fix

I have a branch ready: https://github.com/VANDRANKI/semantic-kernel/tree/fix/redis-return-fields-storage-name

  • _add_return_fields of both collections returns field.storage_name or field.name.
  • The Hashset deserialization decodes the vector in place under the storage name, and also keeps the value under the declared name.

It adds 6 tests to tests/unit/connectors/memory/test_redis_store.py: the RETURN list for both collections with and without vectors, a search through a fake ft.search that honors RETURN, and the vector decode. All 6 fail on main. The 35 tests in that file pass on the branch, and ruff check and format are clean.

I can't open a pull request from this account right now, so the compare link is https://github.com/microsoft/semantic-kernel/compare/main...VANDRANKI:semantic-kernel:fix/redis-return-fields-storage-name

Lenguaje dominante
C#
Estrellas
28.7k
Forks
4.9k
Merge medio
5 d 3 h
PR fusionados (30 d)
63

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/semantic-kernel

Todos los issues de microsoft/semantic-kernel

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.