Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[Feature] Support hybrid search with rrf, weighted_score and mrr rankers

Aberta
  • Algo errado?
#407 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
cpp
Domínio
databases, search

Direção de pesquisa

Start with the Java HybridSearchBuilder and HybridSearchRanker design described in the issue, then inspect the existing table-level full-text search work in #404 and vector-search prerequisite. Implement the rankers, route validation, builder, and snapshot behavior described here, with tests for ranking, ties, weights, empty routes, validation, and snapshot pinning; primary-key support also depends on #410.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

enhancement
Search before asking
  • I searched in the issues and found nothing similar.
Motivation

Sub-issue of #399 (step 4: hybrid search).

Java Table#newHybridSearchBuilder() combines vector and full-text routes into one ranked result. It was built up over several PRs: apache/paimon#8271, apache/paimon#8288, apache/paimon#8294, apache/paimon#8324, apache/paimon#8327, apache/paimon#8336, apache/paimon#8348 and apache/paimon#8351. Paimon C++ has neither a hybrid search API nor a table-level vector search builder.

Java design:

  • Builder. HybridSearchBuilder (HybridSearchBuilderImpl) provides:

    • withPartitionFilter, withFilter
    • addVectorRoute(column, vector, limit[, weight[, options]]); the weight defaults to 1.0
    • addFullTextRoute(field, query, limit, weight)
    • withLimit
    • withRanker(String), withRrfRanker(), withWeightedScoreRanker()
    • executeLocal()

    All routes are pinned to one snapshot, and the filters are forwarded to every route. Full-text routes accept row filters since apache/paimon#9855.

  • Routes. HybridSearchRoute is either VECTOR or FULL_TEXT.

    • Weights must be finite and greater than 0.
    • Full-text routes reject route options.
    • The route builder rejects mixing vector and full-text settings.

  • Rankers. HybridSearchRanker normalizes the ranker name: blank means rrf; otherwise the name is trimmed and lower-cased.

    • rrf: Σ weight / (60 + rank + 1). Ties are broken by score descending, then row id ascending.
    • weighted_score: min-max normalize each route's scores to [0, 1] (all 1.0 when the range is 0), then compute Σ weight × normalized score.
    • mrr: Σ weight / (rank + 1).
    • The final top-k uses a heap.

  • Primary-key tables. They rank physical positions with PrimaryKeySearchRanker.weightedRrf, weightedScore and weightedMrr. Mixing primary-key physical results with global row-id results is rejected.

Solution
  1. Port the rankers and route validation. They are self-contained and can land first with unit tests.
  2. Add the hybrid search builder on top of the table-level full-text search (#404) and a table-level vector search builder.
  3. Add tests:
    • each ranker
    • tie-breaking
    • weights
    • an empty route result
    • route validation
    • snapshot pinning across routes
Anything else?
  • Prerequisite: a table-level vector search builder (Java VectorSearchBuilder / scan / read). It does not exist in Paimon C++ yet, and could be tracked separately or delivered as part of this issue.
  • Depends on #404. Primary-key hybrid search depends on #410.
Are you willing to submit a PR?
  • I'm willing to submit a PR!
Linguagem predominante
C++
Estrelas
65
Forks
31
Merge médio
1d 23h
PRs com merge (30d)
64

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/paimon-cpp

Todas as issues de apache/paimon-cpp

Issues semelhantes

Mais issues de C++

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.