Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[Feature] Support writing ARRAY<BLOB> fields

Aberta
  • Algo errado?
#389 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
cpp, java, python
Domínio
data-engineering, databases

Direção de pesquisa

Start with the ARRAY reader work from #387/#388, then trace BlobFormatWriter, schema validation, the data-evolution write path, and append/data-evolution compaction. Use the BLOB file specification as the format reference. Done means validation, null and descriptor handling, writing, compaction, round-trip tests, and Java/Python compatibility tests all pass.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

enhancement
Search before asking
  • I searched in the issues and found nothing similar.
Motivation

Java Paimon (apache/paimon#8181) and PyPaimon can write top-level ARRAY<BLOB> columns to dedicated .blob files. #387 (PR #388) adds read-only ARRAY<BLOB> support to Paimon C++, but C++ table creation, writes, and append compaction still reject these fields. As a result, C++ users cannot write ordered collections of multimodal objects, such as the frames or images of one sample, to a Paimon table.

Solution

Add Java-compatible write support for top-level ARRAY<BLOB> fields declared with blob-field:

  • Schema validation: Match Java SchemaValidation and ColumnDirectiveUtils:
    • allow ARRAY<BLOB> only as a top-level field declared with blob-field;
    • reject it for blob-descriptor-field, partition keys, and tables without data evolution;
    • reject BLOB elements nested more deeply, such as ARRAY<ARRAY<BLOB>>.
  • Blob format writer: Extend BlobFormatWriter to encode each ARRAY<BLOB> row as the nested payload defined in the BLOB file spec. The payload contains:
    • array magic 1094861634;
    • version 1;
    • element count;
    • concatenated element data;
    • Delta-Varint element length index;
    • index length.
  • Null handling: Distinguish null arrays (outer length -1), empty arrays (element count 0), null elements (element length -1), and empty elements (element length 0).
  • Element sources: Support both inline raw bytes and BlobDescriptor elements. Descriptor elements are materialized from their referenced storage when written, as Java does.
  • Write path: Route ARRAY<BLOB> fields to dedicated blob files in the data-evolution write path, and remove the write-side rejection for them.
  • Compaction: Support ARRAY<BLOB> in append and data-evolution compaction, including placeholder (-2) handling across field groups.
  • Tests:
    • add unit tests for payload encoding, null/empty cases, and error handling;
    • add round-trip tests with the C++ reader from #387;
    • add Java/Python cross-compatibility tests, with C++ writing and Java/Python reading.
Anything else?
  • Depends on #387 / #388 for the reader side.
  • MAP<K, BLOB> write support and ARRAY<BLOB> in primary-key managed BLOB storage (#204) are out of scope and can be tracked separately.
  • No public API changes are expected. The storage format follows the existing Java spec.
Are you willing to submit a PR?
  • I'm willing to submit a PR!
Linguagem predominante
C++
Estrelas
65
Forks
29
Merge médio
2d 5h
PRs com merge (30d)
77

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/paimon-cpp

Todas as issues de apache/paimon-cpp

Issues semelhantes

Mais issues de C++

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.