[C API] Register BFLOAT16 element type in the `StorageSimple` dispatcher

未关闭
  • 哪里不对？
#398 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 5 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
68/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
c, cpp
领域
api, search

调研方向

Start with the dispatcher rows in bindings/c/src/data_builder/simple.hpp, the type mapping in bindings/c/src/types_support.hpp, and the constructor handling in bindings/c/src/storage.hpp. Read the corresponding FLOAT16 tests in bindings/c/tests/ before implementing the BF16 round-trip and index-flow coverage. Done means BF16 storage creation, indexing, searching, saving, loading, and memory reporting work consistently with the stated tolerances.

由索引模型根据 Issue 内容生成。

描述

Summary

SVS_DATA_TYPE_BFLOAT16 is declared in the C API enum
(bindings/c/include/svs/c/svs_c.h), but at pin
5717f6855a73a9ab7524748e2a10a57ed865737d no StorageSimple
specialization is registered for it in the dispatch table at
bindings/c/src/data_builder/simple.hpp:93-104, and
bindings/c/src/storage.hpp:51-55 throws when constructed with
SVS_DATA_TYPE_BFLOAT16. As a result, svs_storage_create_simple(SVS_DATA_TYPE_BFLOAT16, err)
fails, and callers cannot use BF16 vectors as the SVS storage kind.

This ask adds the BFloat16 specialization to the dispatcher so the
type is fully reachable end-to-end.

Motivation

valkey-search supports TYPE FLOAT32, TYPE FLOAT16, and
TYPE BFLOAT16 for its ALGORITHM HNSW index (see valkey-search
PR #1001). Users who already ingest BFloat16 embeddings (common in
recent LLM inference stacks) can pick ALGORITHM HNSW today. For
ALGORITHM SVS_VAMANA, valkey-search rejects TYPE BFLOAT16 at
parse because the SVS C API path cannot store the vectors -- FLOAT32
and FLOAT16 are supported end-to-end but BFLOAT16 is not.

The C++ core supports BFloat16 (svs::BFloat16 exists in
include/svs/lib/bfloat16.h and is used by other paths e.g.
orchestrators/dynamic_ivf.h), so the missing piece is purely a
dispatcher registration in the C API's simple-storage builder.

Proposed change

Add one specialization row to bindings/c/src/data_builder/simple.hpp:

// bindings/c/src/data_builder/simple.hpp:93-104 (approximate)
XX(float,        float_alloc)
XX(svs::Float16, float16_alloc)
XX(svs::BFloat16, bfloat16_alloc)   // <-- new

Plus the corresponding entry in bindings/c/src/types_support.hpp
so to_data_type(svs_data_type_t) maps
SVS_DATA_TYPE_BFLOAT16 to DataType::bfloat16 (currently returns
DataType::undef).

Plus lift the StorageSimple ctor throw at storage.hpp:51-55 for
BFLOAT16.

Acceptance criteria

  1. svs_storage_create_simple(SVS_DATA_TYPE_BFLOAT16, err) returns
    a valid storage handle.
  2. A svs_index_builder_t with a BFloat16 storage builds an index
    via svs_index_build_dynamic. svs_index_dynamic_add_points
    accepts const float* inputs and internally converts to BFloat16
    for storage (mirroring the FLOAT16 path).
  3. svs_index_search_topk on a BFloat16 index returns top-k
    results matching the C++ core's BFloat16 behavior.
  4. svs_storage_get_kind on the storage returns a value consistent
    with the BFloat16 kind. svs_index_get_memory_usage reports
    BFloat16-sized data bytes (2 bytes per element).
  5. Unit tests in bindings/c/tests/:
    • Round-trip: build with BF16 storage, add N vectors, search,
      save, load, search again -- results within numerical
      tolerance appropriate to BFloat16 quantization.
    • Cross-check against the FLOAT16 test at the same N and k --
      confirm the two paths follow the same acceptance shape.

Non-goals

  • Adding BFloat16 as a StorageSQ or StorageLVQ element type.
    Simple-storage (uncompressed) is sufficient; SQ/LVQ inputs are
    independent design decisions.
  • Distance kernel optimization. Scalar fallback is acceptable for
    the initial registration; an AVX / AVX-512-BF16 kernel is a
    separate optimization.
主要语言
C++
星标
241
派生
48
平均合并
4 天 21 小时
30 天内合并 PR
11

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

intel/ScalableVectorSearch 的其他 Issue

查看 intel/ScalableVectorSearch 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。