[C API] Register BFLOAT16 element type in the `StorageSimple` dispatcher

Ouverte
  • Quelque chose ne va pas ?
#398 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 5 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
68/100
Type d'issue
Fonctionnalité
Clarté
Clairement spécifiée
Activité
Active
Stack technique
c, cpp
Domaine
api, search

Piste de recherche

Start with the dispatcher rows in bindings/c/src/data_builder/simple.hpp, the type mapping in bindings/c/src/types_support.hpp, and the constructor handling in bindings/c/src/storage.hpp. Read the corresponding FLOAT16 tests in bindings/c/tests/ before implementing the BF16 round-trip and index-flow coverage. Done means BF16 storage creation, indexing, searching, saving, loading, and memory reporting work consistently with the stated tolerances.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Summary

SVS_DATA_TYPE_BFLOAT16 is declared in the C API enum
(bindings/c/include/svs/c/svs_c.h), but at pin
5717f6855a73a9ab7524748e2a10a57ed865737d no StorageSimple
specialization is registered for it in the dispatch table at
bindings/c/src/data_builder/simple.hpp:93-104, and
bindings/c/src/storage.hpp:51-55 throws when constructed with
SVS_DATA_TYPE_BFLOAT16. As a result, svs_storage_create_simple(SVS_DATA_TYPE_BFLOAT16, err)
fails, and callers cannot use BF16 vectors as the SVS storage kind.

This ask adds the BFloat16 specialization to the dispatcher so the
type is fully reachable end-to-end.

Motivation

valkey-search supports TYPE FLOAT32, TYPE FLOAT16, and
TYPE BFLOAT16 for its ALGORITHM HNSW index (see valkey-search
PR #1001). Users who already ingest BFloat16 embeddings (common in
recent LLM inference stacks) can pick ALGORITHM HNSW today. For
ALGORITHM SVS_VAMANA, valkey-search rejects TYPE BFLOAT16 at
parse because the SVS C API path cannot store the vectors -- FLOAT32
and FLOAT16 are supported end-to-end but BFLOAT16 is not.

The C++ core supports BFloat16 (svs::BFloat16 exists in
include/svs/lib/bfloat16.h and is used by other paths e.g.
orchestrators/dynamic_ivf.h), so the missing piece is purely a
dispatcher registration in the C API's simple-storage builder.

Proposed change

Add one specialization row to bindings/c/src/data_builder/simple.hpp:

// bindings/c/src/data_builder/simple.hpp:93-104 (approximate)
XX(float,        float_alloc)
XX(svs::Float16, float16_alloc)
XX(svs::BFloat16, bfloat16_alloc)   // <-- new

Plus the corresponding entry in bindings/c/src/types_support.hpp
so to_data_type(svs_data_type_t) maps
SVS_DATA_TYPE_BFLOAT16 to DataType::bfloat16 (currently returns
DataType::undef).

Plus lift the StorageSimple ctor throw at storage.hpp:51-55 for
BFLOAT16.

Acceptance criteria

  1. svs_storage_create_simple(SVS_DATA_TYPE_BFLOAT16, err) returns
    a valid storage handle.
  2. A svs_index_builder_t with a BFloat16 storage builds an index
    via svs_index_build_dynamic. svs_index_dynamic_add_points
    accepts const float* inputs and internally converts to BFloat16
    for storage (mirroring the FLOAT16 path).
  3. svs_index_search_topk on a BFloat16 index returns top-k
    results matching the C++ core's BFloat16 behavior.
  4. svs_storage_get_kind on the storage returns a value consistent
    with the BFloat16 kind. svs_index_get_memory_usage reports
    BFloat16-sized data bytes (2 bytes per element).
  5. Unit tests in bindings/c/tests/:
    • Round-trip: build with BF16 storage, add N vectors, search,
      save, load, search again -- results within numerical
      tolerance appropriate to BFloat16 quantization.
    • Cross-check against the FLOAT16 test at the same N and k --
      confirm the two paths follow the same acceptance shape.

Non-goals

  • Adding BFloat16 as a StorageSQ or StorageLVQ element type.
    Simple-storage (uncompressed) is sufficient; SQ/LVQ inputs are
    independent design decisions.
  • Distance kernel optimization. Scalar fallback is acceptable for
    the initial registration; an AVX / AVX-512-BF16 kernel is a
    separate optimization.
Langage dominant
C++
Étoiles
241
Forks
48
Merge moyen
4 j 21 h
PR mergées (30 j)
11

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de intel/ScalableVectorSearch

Toutes les issues de intel/ScalableVectorSearch

Issues similaires

Plus d'issues C++

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.