Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

bug: manifest Avro headers use the wrong partition-spec shape and omit schema-id

Fechada
  • Algo errado?
#1,016 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 2 dias

@mavam já está trabalhando nisso.

Desde 14/10/2026.

  • #1017 de @mavam — aberto

Avaliação

Dificuldade
4/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
58/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
cpp
Domínio
data-engineering

Direção de pesquisa

Start in src/iceberg/manifest/v2_metadata.cc, at the Init() method of the v2 adapter, where partition-spec is serialized from the whole PartitionSpec and schema-id is never set. Check the same pattern in v1_metadata.cc and v3_metadata.cc. Use the fastavro script from the issue to inspect headers of a written manifest. Done when the partition-spec header holds only the partition-fields array, and v2/v3 manifests carry a schema-id header equal to the current schema ID, including nonzero IDs.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Summary

Manifest Avro headers written by iceberg-cpp do not conform to the Iceberg manifest metadata contract:

  1. The partition-spec header contains the full partition-spec object instead of only its partition-fields array.
  2. The separate schema-id header is omitted, although it is required for v2 manifests.

The first defect prevents ClickHouse from reading the resulting table, including queries that select only scalar fields. Both defects occur at the manifest metadata adapter boundary, independently of the data schema or Parquet contents.

Versions and scope

  • Observed in generated v2 manifests with iceberg-cpp commit 131f9763ecca8b5985e48bb06a957b4bebc6779f, used by Tenzir's to_iceberg writer.
  • Confirmed by source inspection to remain in main at 57ec6b8101952a47916ab9cd4328a17319b40195.
  • The completed interoperability test used an unpartitioned v2 table created through apache/iceberg-rest-fixture:1.10.1, followed by four sequential appends through NewFastAppend(). All four original manifests had both header defects.
  • The incorrect partition-spec serialization also appears in the v1 and v3 adapter implementations. These versions were inspected, not exercised in the interoperability test.

Expected and actual metadata

The Iceberg 1.10.1 manifest specification requires:

  • partition-spec: "JSON representation of only the partition fields array of the partition spec used to write the manifest."
  • schema-id: "ID of the schema used to write the manifest as a string." This header is optional in v1 and required in v2.

For an unpartitioned manifest written with schema ID 0, the expected headers are:

partition-spec = []
schema-id = 0

The actual headers are:

partition-spec = {"fields":[],"spec-id":0}
schema-id = <absent>

The JSON schema header contains its own schema-id, but this does not supply the separately required Avro key-value metadata entry. In our four manifests, the embedded schema IDs were 0, 0, 1, and 2; the separate header was absent in every case.

Inspect an affected manifest

After creating an unpartitioned v2 table and appending a data file through iceberg-cpp, inspect the generated data manifest, not its manifest list:

uv run --no-project --with fastavro python - /path/to/data-manifest.avro <<'PY'
import json
import sys
from fastavro import reader

with open(sys.argv[1], "rb") as stream:
    metadata = reader(stream).metadata

print("partition-spec:", metadata["partition-spec"])
print("schema-id:", metadata.get("schema-id"))
print("embedded schema-id:", json.loads(metadata["schema"])["schema-id"])
PY

Observed output for the first original manifest:

partition-spec: {"fields":[],"spec-id":0}
schema-id: None
embedded schema-id: 0

Source of the defects

The v2 adapter's Init() implementation calls the general serializer for the entire PartitionSpec:

ICEBERG_ASSIGN_OR_RAISE(metadata_["schema"], ToJsonString(*current_schema_))
ICEBERG_ASSIGN_OR_RAISE(metadata_["partition-spec"], ToJsonString(*partition_spec_));
metadata_["partition-spec-id"] = std::to_string(partition_spec_->spec_id());
metadata_["format-version"] = "2";

There is no assignment for metadata_["schema-id"].

The same serialization pattern appears in ManifestEntryAdapterV1::Init() and ManifestEntryAdapterV3::Init().

Interoperability impact

ClickHouse-local 26.7.1.1315 and 26.9.17.8 failed actual data scans with Poco::Bad cast exception while parsing the object-valued partition-spec header as a JSON array. This affected scalar projections such as class_uid, so it was not a nested-Parquet limitation. Optimized count() queries could still return metadata-derived counts and did not establish data readability.

For diagnosis only, replacing the header with [] in disposable manifests and updating their recorded manifest_length in the manifest lists removed this parsing failure. Parsed manifest records and schemas remained unchanged. This was not a production repair or a source-code fix.

PyIceberg tolerated the original headers and read the expected data. Its acceptance does not establish specification conformance. We did not isolate a reader failure attributable specifically to the missing schema-id header.

Suggested fix and regression coverage

Use manifest-specific serialization for the partition-fields array; retain the general partition-spec object serializer for REST and table metadata contexts where that shape is correct. Emit the separate schema ID from current_schema_->schema_id().

Add assertions against the actual written Avro headers for partitioned and unpartitioned manifests across supported format versions. Cover nonzero schema IDs as well as the initial schema, and check that v2/v3 manifests include the separately required schema-id entry. An independent-reader interoperability regression would help prevent permissive readers or writer/reader round trips from hiding these defects.

Linguagem predominante
C++
Estrelas
228
Forks
136
Merge médio
5d 19h
PRs com merge (30d)
53

Preparar o ambiente

Este projeto não oferece contêiner de desenvolvimento, Dockerfile nem guia de contribuição, então a configuração fica por sua conta: comece pelo README e veja nosso guia da primeira contribuição para os passos gerais.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/iceberg-cpp

Todas as issues de apache/iceberg-cpp

Issues semelhantes

Mais issues de C++

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.