Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Make manifest records schema-aware to avoid field mismatches across format versions

Aberta
  • Algo errado?
#3,957 0 comentários 0 reações 1 responsável Ver no GitHub

Mantenedores costumam responder em até 1 dia

@kevinjqliu já está trabalhando nisso.

Desde 12/9/2026.

Avaliação

Esta issue ainda não foi avaliada.

Descrição

bug
Apache Iceberg version

main (development), commit 308768d99

Please describe the bug 🐞

Manifest records do not retain their schema, and their property getters use hard-coded field positions. When the in-memory record layout differs from the file schema, this can cause incorrect field access or silent data loss during serialization.

For example, referenced_data_file has field ID 143 in both v2 and v3, but its zero-based position differs:

Field v2 position v3 position
first_row_id Not present 16
referenced_data_file 16 17

This causes two problems:

  • Writing a v3 record to a v2 file: DataFile.from_args() defaults to the v3 layout. When writing it through AvroOutputFile with a v2 file schema, omitting record_schema makes the writer read position 16 instead of 17 for referenced_data_file. If first_row_id is null, a non-null reference is silently written as null. Explicitly supplying the v3 record schema enables the existing field-ID projection and preserves the value.
  • Accessing a v2 record: DataFile.from_args(_table_format_version=2, ...) stores the reference at position 16, but the getters still assume v3 positions. Consequently, first_row_id returns the reference path and referenced_data_file raises IndexError. Reading a v2 file into DataFile without projecting to v3 has the same mismatch.

The normal manifest read/write helpers already supply the v3 projection. However, correctness still depends on callers separately tracking the in-memory layout and supplying the matching schema.

Proposed behavior

Records should keep the schema that describes their in-memory layout. Getters should use field IDs, not assume a particular version’s field positions.

When reading, use the schema from the file. If the read projects into another schema, retain that schema on the resulting record.

When writing, callers should only need to specify the target format version, or file_schema when using AvroOutputFile. IO should determine the input layout from the record and handle the conversion before encoding. This needs to account for nested records and partition fields, not just the top-level record.

We shouldn’t need to pass record_schema just to prevent a field from being read at the wrong position. Keep it supported for callers that explicitly provide it.

For the example above, writing a default v3 record to a v2 file should preserve referenced_data_file without passing record_schema. If a conversion isn’t supported, raise an error rather than silently writing the wrong value.

Willingness to contribute

N/A

Linguagem predominante
Python
Estrelas
1.1k
Forks
606
Merge médio
1d 11h
PRs com merge (30d)
75

Preparar o ambiente

  • Sem Dockerfile nem arquivo Docker Compose
  • Tem um modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/iceberg-python

Todas as issues de apache/iceberg-python

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.