New index building APIs for simpler and flexible usage

未关闭
  • 哪里不对？
#189 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 5 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
30/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
停滞
技术栈
cpp
领域
api, search

调研方向

首先查看现有的 Vamana::build API 以及 issue 中描述的 SQDataset::compress 流程。此变更应提供接受未压缩数据和数据集类型的索引构建入口，支持回退到未压缩数据，并允许压缩和图构建独立进行。

由索引模型根据 Issue 内容生成。

描述

enhancement

In SVS, we separate the concept of dataset and index. This allows an index to accept different datasets, such as FP32, Scalar Quantization (SQDataset), LVQ, and LeanVec.
Currently, using SQDataset as an example, users must call compress to get a SQDataset, then pass it to the index. However, most use cases don't care about the dataset itself - this two-step process is often unnecessary for most users. Additionally, this approach makes runtime fallback impossible to implement in SVS, as each dataset (i.e., type) is determined at compile time.

auto loaded =
        svs::VectorDataLoader<float>(std::filesystem::path(SVS_DATA_DIR) / "data_f32.svs").load();
auto data =
        svs::scalar::SQDataset<std::int8_t>::compress(loaded, threadpool); // SQDataset is determined at compile time
auto parameters = svs::index::vamana::VamanaBuildParameters{};
svs::Vamana index = svs::Vamana::build<float>(
        parameters, data, svs::distance::DistanceL2(), num_threads
);

I propose adding index building APIs that directly accept uncompressed data and take the dataset type as a parameter to determine which dataset format to use internally.

auto loaded =
        svs::VectorDataLoader<float>(std::filesystem::path(SVS_DATA_DIR) / "data_f32.svs").load();
auto parameters = svs::index::vamana::VamanaBuildParameters{};
svs::Vamana index = svs::Vamana::build<float>(
        parameters, data, svs::distance::DistanceL2(), num_threads, svs::SQ8
); // internally, SVS fallbacks to uncompressed data if scalar quantization failed

Another advantage of this is that we could optionally utilize uncompressed data to build the graph rather than compressed data, which typically gives worse quality graphs as compression introduces approximation errors that can degrade the graph structure during construction.
Note that when calling build, we can compress the data and build the graph in parallel, as the two things are independent.

主要语言
C++
星标
241
派生
48
平均合并
4 天 21 小时
30 天内合并 PR
11

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

intel/ScalableVectorSearch 的其他 Issue

查看 intel/ScalableVectorSearch 的全部 Issue

相似的 Issue

更多 C++ Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。