Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[Feature] Support richer data types, starting with VECTOR<T, N> based on PIP-40

Aberta
  • Algo errado?
#197 0 comentários 0 reações 1 responsável Ver no GitHub

Mantenedores costumam responder em até 1 dia

@ChaomingZhangCN já está trabalhando nisso.

Desde 12/8/2026.

Avaliação

Esta issue ainda não foi avaliada.

Descrição

enhancement
Search before asking
  • I searched in the issues and found nothing similar.
Motivation

Apache Paimon Java has introduced the VECTOR<T, N> data type based on PIP-40. It supports schema representation, regular data-file storage such as Parquet, dedicated vector storage, reads, writes, and Data Evolution.

Paimon C++ currently recognizes .vector. file names for some file-level bookkeeping, but it does not yet provide a VECTOR logical type, Arrow mapping, serialization, storage, or end-to-end read and write support.

This issue implements the VECTOR roadmap item tracked in #186.

Solution

Introduce VECTOR<T, N> support incrementally, while keeping the schema and storage behavior compatible with Apache Paimon Java.

Phase 1: Schema and regular Parquet storage
  • Add VECTOR<T, N> to the Paimon C++ logical type system.
  • Implement schema JSON serialization and deserialization compatible with Paimon Java.
  • Map VECTOR<T, N> to Arrow FixedSizeList<T, N>.
  • Support the element types defined by PIP-40:
    • BOOLEAN
    • TINYINT
    • SMALLINT
    • INT
    • BIGINT
    • FLOAT
    • DOUBLE
  • Validate that the dimension is positive and fixed.
  • Validate that the written vector length equals N.
  • Reject null vector elements.
  • Support reading and writing VECTOR columns in regular Parquet data files.
  • Add end-to-end append-table tests.
  • Add Java/C++ schema and file compatibility tests.
Phase 2: Schema Evolution, Data Evolution, and Primary-Key Table Baseline
Schema Evolution
  • Support adding and dropping VECTOR columns.
  • Support reading files written with previous table schemas.
  • Preserve VECTOR values when projecting files across schema versions.
  • Reject incompatible dimension changes, such as
    VECTOR<FLOAT, 3> to VECTOR<FLOAT, 5>.
  • Reject incompatible element-type changes.
  • Add schema-evolution integration tests for both append-only and
    supported primary-key tables.
Data Evolution Read/Write
  • Support VECTOR columns in row-tracking append-only tables with
    data-evolution.enabled = true.
  • Support full-row writes containing VECTOR columns.
  • Support partial-column writes containing VECTOR columns through the
    write-schema path.
  • Merge VECTOR values from files covering the same row-id range.
  • Support reading VECTOR columns across different file schema IDs.
  • Add Data Evolution integration tests covering full writes,
    partial VECTOR writes, null values, and mixed old/new schema files.
Primary-Key Table Baseline
  • Allow VECTOR columns as ordinary non-key value columns in
    deduplicate primary-key tables backed by regular Parquet files.
  • Support insert, same-key update, and read.
  • Support write-buffer spill and compaction.
  • Support adding and dropping VECTOR value columns while retaining
    readability of files written with previous schemas.
  • Reject VECTOR columns as primary, partition, bucket, sequence,
    sequence-group ordering, or sorting fields.
  • Explicitly reject lookup, partial-update, aggregation, and other
    unsupported primary-key configurations until they are implemented.
  • Add end-to-end primary-key table tests for write, update, read,
    spill, compaction, and schema evolution.
Phase 3: Dedicated vector storage
  • Support the vector file format configuration used by Paimon Java.
  • Support reading and writing dedicated *.vector.vortex files.
  • Integrate vector files with row tracking and Data Evolution.
  • Integrate vector files with scan planning, file commits, and conflict handling.
  • Add Java, Python, and C++ Vortex compatibility tests.
Initial scope

The initial implementation can focus on Phase 1, providing a usable end-to-end vertical slice through schema representation, Arrow mapping, and regular Parquet reads and writes.

Dedicated Vortex vector storage and Data Evolution can be delivered through follow-up pull requests under this issue.

The following items are not required for the initial implementation:

  • ORC VECTOR support
  • Vector indexes or similarity search
  • Changing vector dimensions through schema evolution
  • VECTOR values inside shared-shredding MAP columns
  • Element types not supported by Apache Paimon Java

PIP-40 should only be considered fully supported after all phases are complete. Completing Phase 1 means that regular Parquet VECTOR storage is supported, but does not imply support for dedicated Vortex vector files.

Anything else?

Related roadmap: #186

Are you willing to submit a PR?
  • I'm willing to submit a PR!
Linguagem predominante
C++
Estrelas
65
Forks
31
Merge médio
1d 23h
PRs com merge (30d)
64

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/paimon-cpp

Todas as issues de apache/paimon-cpp

Issues semelhantes

Mais issues de C++

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.