bug: manifest Avro headers use the wrong partition-spec shape and omit schema-id

Start in src/iceberg/manifest/v2_metadata.cc, at the Init() method of the v2 adapter, where partition-spec is serialized from the whole PartitionSpec and schema-id is never set. Check the same pattern in v1_metadata.cc and v3_metadata.cc. Use the fastavro script from the issue to inspect headers of a written manifest. Done when the partition-spec header holds only the partition-fields array, and v2/v3 manifests carry a schema-id header equal to the current schema ID, including nonzero IDs.

Summary

Manifest Avro headers written by iceberg-cpp do not conform to the Iceberg manifest metadata contract:

1. The partition-spec header contains the full partition-spec object instead of only its partition-fields array.
2. The separate schema-id header is omitted, although it is required for v2 manifests.

The first defect prevents ClickHouse from reading the resulting table, including queries that select only scalar fields. Both defects occur at the manifest metadata adapter boundary, independently of the data schema or Parquet contents.

Versions and scope

• Observed in generated v2 manifests with iceberg-cpp commit 131f9763ecca8b5985e48bb06a957b4bebc6779f, used by Tenzir's to_iceberg writer.
• Confirmed by source inspection to remain in main at 57ec6b8101952a47916ab9cd4328a17319b40195.
• The completed interoperability test used an unpartitioned v2 table created through apache/iceberg-rest-fixture:1.10.1, followed by four sequential appends through NewFastAppend(). All four original manifests had both header defects.
• The incorrect partition-spec serialization also appears in the v1 and v3 adapter implementations. These versions were inspected, not exercised in the interoperability test.

Expected and actual metadata

The Iceberg 1.10.1 manifest specification requires:

• partition-spec: "JSON representation of only the partition fields array of the partition spec used to write the manifest."
• schema-id: "ID of the schema used to write the manifest as a string." This header is optional in v1 and required in v2.

For an unpartitioned manifest written with schema ID 0, the expected headers are:

partition-spec = []
schema-id = 0

The actual headers are:

partition-spec = {"fields":[],"spec-id":0}
schema-id = <absent>

The JSON schema header contains its own schema-id, but this does not supply the separately required Avro key-value metadata entry. In our four manifests, the embedded schema IDs were 0, 0, 1, and 2; the separate header was absent in every case.

Inspect an affected manifest

After creating an unpartitioned v2 table and appending a data file through iceberg-cpp, inspect the generated data manifest, not its manifest list:

uv run --no-project --with fastavro python - /path/to/data-manifest.avro <<'PY'
import json
import sys
from fastavro import reader

with open(sys.argv[1], "rb") as stream:
    metadata = reader(stream).metadata

print("partition-spec:", metadata["partition-spec"])
print("schema-id:", metadata.get("schema-id"))
print("embedded schema-id:", json.loads(metadata["schema"])["schema-id"])
PY

Observed output for the first original manifest:

partition-spec: {"fields":[],"spec-id":0}
schema-id: None
embedded schema-id: 0

Source of the defects

The v2 adapter's Init() implementation calls the general serializer for the entire PartitionSpec:

ICEBERG_ASSIGN_OR_RAISE(metadata_["schema"], ToJsonString(*current_schema_))
ICEBERG_ASSIGN_OR_RAISE(metadata_["partition-spec"], ToJsonString(*partition_spec_));
metadata_["partition-spec-id"] = std::to_string(partition_spec_->spec_id());
metadata_["format-version"] = "2";

There is no assignment for metadata_["schema-id"].

The same serialization pattern appears in ManifestEntryAdapterV1::Init() and ManifestEntryAdapterV3::Init().

Interoperability impact

ClickHouse-local 26.7.1.1315 and 26.9.17.8 failed actual data scans with Poco::Bad cast exception while parsing the object-valued partition-spec header as a JSON array. This affected scalar projections such as class_uid, so it was not a nested-Parquet limitation. Optimized count() queries could still return metadata-derived counts and did not establish data readability.

For diagnosis only, replacing the header with [] in disposable manifests and updating their recorded manifest_length in the manifest lists removed this parsing failure. Parsed manifest records and schemas remained unchanged. This was not a production repair or a source-code fix.

PyIceberg tolerated the original headers and read the expected data. Its acceptance does not establish specification conformance. We did not isolate a reader failure attributable specifically to the missing schema-id header.

Suggested fix and regression coverage

Use manifest-specific serialization for the partition-fields array; retain the general partition-spec object serializer for REST and table metadata contexts where that shape is correct. Emit the separate schema ID from current_schema_->schema_id().

Add assertions against the actual written Avro headers for partitioned and unpartitioned manifests across supported format versions. Cover nonzero schema IDs as well as the initial schema, and check that v2/v3 manifests include the separately required schema-id entry. An independent-reader interoperability regression would help prevent permissive readers or writer/reader round trips from hiding these defects.