avro schema reader ignores iceberg-field-name property — returns sanitized names from java manifests

Open Beginner friendly
#2,536 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
rust
Domain
data-engineering

Research direction

Start in crates/iceberg/src/avro/schema.rs, focusing on avro_schema_to_schema and how each Avro field's name is read. Compare the documented iceberg-field-name property with Java's ICEBERG_FIELD_NAME_PROP convention, then verify that Java-written manifests expose the original partition field name rather than the sanitized Avro name.

Written by the indexing model from the issue text.

Description

problem

when java writes manifests with partition field names that violate avro naming rules, it sanitizes them (e.g. 815d3b... → _815d3b...) and stores the original in an iceberg-field-name avro field property. iceberg-rust's avro reader uses the avro field name directly without checking this property, so it returns the sanitized name instead of the original.

this causes field name mismatches when reading java-written manifests — the partition field name in the manifest entry won't match the partition field name in the table's partition spec.

relevant code

crates/iceberg/src/avro/schema.rs — avro_schema_to_schema reads &avro_field.name without checking for iceberg-field-name custom property.

java's read path resolves fields by field-id (integer), but also provides the original name via ICEBERG_FIELD_NAME_PROP:

public static final String ICEBERG_FIELD_NAME_PROP = "iceberg-field-name";

reproduction

  1. create a table in java with a partition field name starting with a digit
  2. write data (java produces manifests with sanitized avro field names + iceberg-field-name property)
  3. read the manifest in iceberg-rust — field names will be the sanitized form, not the original

expected behavior

when reading avro schemas from manifests, check each field for the iceberg-field-name property. if present, use that as the iceberg field name instead of the avro field name.

notes

  • iceberg resolves fields by field-id in most paths, so the impact is limited to code that uses field names from manifest entries
  • this is a well-established convention in the java implementation (since 2019)
  • pyiceberg implements its own avro reader that bypasses name validation entirely
Dominant language
Rust
Stars
1.4k
Forks
574
Avg merge
1d 18h
Merged PRs (30d)
84

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg-rust

All issues in apache/iceberg-rust

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.