Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[Feature] Support vector and full-text search in File Index

Aberta
  • Algo errado?
#361 0 comentários 0 reações 1 responsável Ver no GitHub

Mantenedores costumam responder em até 1 dia

@zjw1111 já está trabalhando nisso.

Desde 21/9/2026.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
45/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
cpp
Domínio
data-engineering, search

Direção de pesquisa

Start with the existing File Index write path, container format, readers, and the work from #173. Trace how read-context requests reach FileIndexReader and how selected row positions reach the data-file reader. Done means vector and full-text indexes round-trip through write/read paths, support the listed edge cases, and preserve per-file TopK score alignment.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

enhancement
Search before asking
  • I searched in the issues and found nothing similar.
Motivation

Paimon C++ has File Index infrastructure for per-data-file indexes, but does not yet provide an end-to-end File Index search path for vector similarity or full-text queries. Applications should be able to build these indexes with their data files and use them to select matching file-local rows without requiring a Global Index.

Solution

Add Vector Search and Full Text Search support to File Index:

  • Build and serialize vector and full-text index artifacts through the existing File Index write path and container format, and open them through File Index readers.
  • Add file-local search visits to FileIndexReader and route search requests from the read context to the appropriate index reader.
  • Apply returned file-local row positions to the data-file reader. Vector Search should return a TopK result independently for each data file and keep scores aligned with selected rows; expose scores through an explicitly projected _INDEX_SCORE column.
  • Support Full Text Search row selection. Score output for Full Text Search can be handled separately.
  • Add focused tests for write/read round trips, null or empty indexes, multiple data files and batches, score alignment, and columns with more than one File Index type.

The initial implementation can target append-only/raw-file reads. Predicate combination, deletion-vector and row-range pre-filtering, Merge-on-Read/Data Evolution, and cross-file global TopK should be handled in follow-up work so that filtering does not silently change TopK semantics.

Anything else?

This builds on the File Index writing work in #173; it is specifically about vector/full-text index implementations and their search read path.

Are you willing to submit a PR?
  • I'm willing to submit a PR!
Linguagem predominante
C++
Estrelas
65
Forks
31
Merge médio
2d 3h
PRs com merge (30d)
73

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/paimon-cpp

Todas as issues de apache/paimon-cpp

Issues semelhantes

Mais issues de C++

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.