Make manifest records schema-aware to avoid field mismatches across format versions

Apache Iceberg version

main (development), commit 308768d99

Please describe the bug 🐞

Manifest records do not retain their schema, and their property getters use hard-coded field positions. When the in-memory record layout differs from the file schema, this can cause incorrect field access or silent data loss during serialization.

For example, referenced_data_file has field ID 143 in both v2 and v3, but its zero-based position differs:

This causes two problems:

• Writing a v3 record to a v2 file: DataFile.from_args() defaults to the v3 layout. When writing it through AvroOutputFile with a v2 file schema, omitting record_schema makes the writer read position 16 instead of 17 for referenced_data_file. If first_row_id is null, a non-null reference is silently written as null. Explicitly supplying the v3 record schema enables the existing field-ID projection and preserves the value.
• Accessing a v2 record: DataFile.from_args(_table_format_version=2, ...) stores the reference at position 16, but the getters still assume v3 positions. Consequently, first_row_id returns the reference path and referenced_data_file raises IndexError. Reading a v2 file into DataFile without projecting to v3 has the same mismatch.

The normal manifest read/write helpers already supply the v3 projection. However, correctness still depends on callers separately tracking the in-memory layout and supplying the matching schema.

Proposed behavior

Records should keep the schema that describes their in-memory layout. Getters should use field IDs, not assume a particular version’s field positions.

When reading, use the schema from the file. If the read projects into another schema, retain that schema on the resulting record.

When writing, callers should only need to specify the target format version, or file_schema when using AvroOutputFile. IO should determine the input layout from the record and handle the conversion before encoding. This needs to account for nested records and partition fields, not just the top-level record.

We shouldn’t need to pass record_schema just to prevent a field from being read at the wrong position. Keep it supported for callers that explicitly provide it.

For the example above, writing a default v3 record to a v2 file should preserve referenced_data_file without passing record_schema. If a conversion isn’t supported, raise an error rather than silently writing the wrong value.

Willingness to contribute

N/A