Python: Bug: Redis search returns None for every field that declares a storage_name, because RETURN lists the declared name

Bắt đầu tại python/semantic_kernel/connectors/redis.py, đọc _add_return_fields trong cả hai lớp collection và RedisHashsetCollection._deserialize_store_models_to_dicts. Sau đó kiểm tra tests/unit/connectors/memory/test_redis_store.py và chạy các bài kiểm thử của tệp này. Công việc hoàn tất khi cả hai collection trả về tên lưu trữ trong kết quả tìm kiếm và quá trình giải tuần tự hóa vector của Hashset tạo ra một danh sách số thực cho cả thao tác tìm kiếm lẫn lấy dữ liệu.

Description

When a field of a Redis collection declares a storage_name, a search returns the record without that field.

The collection creates the index attributes under storage_name or name, and it stores the values under storage_name or name. The KNN part of the search query also uses the storage name. But the RETURN clause of the query lists the declared field name. The server has no attribute with that name, so it leaves the field out.

• A data field with a storage_name comes back as None (dict or dataclass record types). For a pydantic record type, the validation fails because the field is missing.
• With include_vectors=True, a vector field with a storage_name is not returned either.
• RedisHashsetCollection has a second problem in the same area. It decodes the vector bytes and writes the array under the declared name. The data model step then reads the storage name key, so the vector stays raw bytes. This also hits get, not only search.

Both RedisHashsetCollection and RedisJsonCollection have the wrong RETURN list.

Reproduction

I ran this on current main (df1cb33d0), Python 3.13, redis-py and redisvl 0.27.2. I have no Redis Stack server here. The fake ft(...).search below answers the way RediSearch does for RETURN: only the attributes named in RETURN come back, plus the key. The RETURN list it prints is the one the connector sends.

import asyncio
import struct
import warnings
from types import SimpleNamespace

warnings.filterwarnings("ignore")
from redis.asyncio import Redis
from redis.commands.search.document import Document
from redisvl.schema import IndexSchema

import semantic_kernel.connectors.redis as redis_module
from semantic_kernel.connectors.redis import RedisHashsetCollection
from semantic_kernel.data.vector import VectorStoreCollectionDefinition, VectorStoreField

DEFN = VectorStoreCollectionDefinition(
    fields=[
        VectorStoreField("key", name="id", type="str"),
        VectorStoreField("data", name="title", type="str", storage_name="doc_title", is_indexed=True),
        VectorStoreField("vector", name="vec", dimensions=3, storage_name="doc_vec", type="float"),
    ]
)
VEC = struct.pack("<3f", 0.1, 0.2, 0.3)
STORED_HASH = {"doc_title": "hello", "doc_vec": VEC, "vector_distance": "0.25"}


class FakeFT:
    async def search(self, query, query_params=None):
        args = query.get_args()
        i = args.index("RETURN")
        wanted = args[i + 2 : i + 2 + args[i + 1]]
        print("RETURN sent to the server:", wanted)
        doc = {"id": "c:k1", "payload": None}
        for name in wanted:
            if name in STORED_HASH:
                doc[name] = STORED_HASH[name]
        return SimpleNamespace(docs=[Document(**doc)], total=1)


SCHEMA = IndexSchema.from_dict(
    {
        "index": {"name": "c", "prefix": "c", "storage_type": "hash"},
        "fields": [
            {"name": "doc_title", "type": "text"},
            {
                "name": "doc_vec",
                "type": "vector",
                "attrs": {"dims": 3, "algorithm": "flat", "distance_metric": "cosine", "datatype": "float32"},
            },
        ],
    }
)


async def fake_from_existing(*args, **kwargs):
    return SimpleNamespace(schema=SCHEMA)


redis_module.AsyncSearchIndex.from_existing = fake_from_existing


async def main():
    db = Redis()
    db.ft = lambda name: FakeFT()
    col = RedisHashsetCollection(record_type=dict, definition=DEFN, collection_name="c", redis_database=db)
    print("stored attributes:", ["doc_title", "doc_vec"])
    res = await col.search(vector=[0.1, 0.2, 0.3], top=3, include_vectors=True)
    async for r in res.results:
        print("search record:", r.record)

    # the get path: a stored hash read back through the same deserialization
    store_record = {"id": "k1", "doc_title": "hello", "doc_vec": VEC}
    print("get record:", col.deserialize([store_record], include_vectors=True))


asyncio.run(main())

Output on main:

stored attributes: ['doc_title', 'doc_vec']
RETURN sent to the server: ['vector_distance', 'id', 'title', 'vec']
search record: {'id': 'c:k1', 'title': None, 'vec': None}
get record: [{'id': 'k1', 'title': 'hello', 'vec': b'\xcd\xcc\xcc=\xcd\xccL>\x9a\x99\x99>'}]

The stored hash has doc_title and doc_vec, but the query asks for title and vec. The search result has None for both fields. The get record has the title, but the vector is still bytes.

For RedisJsonCollection the generated RETURN list is the same, ['vector_distance', 'id', 'title', 'vec'], while the index attributes are doc_title and doc_vec. I printed it with col._construct_vector_query([0.1, 0.2, 0.3], VectorSearchOptions(include_vectors=True)) and query.get_args().

Cause

In python/semantic_kernel/connectors/redis.py, _add_return_fields of both collections calls query.return_field(field.name). Everything else in the file uses field.storage_name or field.name: _definition_to_redis_fields, the serializers, the KNN vector_field_name and the filter parser.

In RedisHashsetCollection._deserialize_store_models_to_dicts, the vector is decoded into rec[field.name]. VectorStoreCollection._deserialize_dict_to_data_model reads record.get(field.storage_name or field.name) for dict and dataclass types, and for pydantic types it moves the storage key over the declared name. In both cases the raw bytes win.

The same thing was reported and fixed for Azure AI Search hybrid search in #14684.

Expected

A field with a storage_name is returned by search and by get, with the same value as a field without one. The vector is decoded to a list of floats.

Possible fix

I have a branch ready: https://github.com/VANDRANKI/semantic-kernel/tree/fix/redis-return-fields-storage-name

• _add_return_fields of both collections returns field.storage_name or field.name.
• The Hashset deserialization decodes the vector in place under the storage name, and also keeps the value under the declared name.

It adds 6 tests to tests/unit/connectors/memory/test_redis_store.py: the RETURN list for both collections with and without vectors, a search through a fake ft.search that honors RETURN, and the vector decode. All 6 fail on main. The 35 tests in that file pass on the branch, and ruff check and format are clean.

I can't open a pull request from this account right now, so the compare link is https://github.com/microsoft/semantic-kernel/compare/main...VANDRANKI:semantic-kernel:fix/redis-return-fields-storage-name