Cache manifest and manifest-list bytes by location, as Iceberg Java's `io.manifest.cache-*` does

Aberta
  • Algo errado?
#4,300 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

@Anurag-S1ngh já está trabalhando nisso.

Desde 11/11/2026.

  • #4335 de @Anurag-S1ngh — aberto

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
python
Domínio
databases

Direção de pesquisa

Start by tracing load_table, plan_files, _manifests, and fetch_manifest_entry to confirm where metadata, manifest-list, and manifest bytes are read. Then review the existing _manifest_cache and the related issues #1187, #3978, and #3991. Done means callers can supply a bounded, location-keyed byte cache for manifests and manifest lists, disabled by default, while parsing remains unchanged.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Feature Request / Improvement

Planning the same table repeatedly reads the same immutable files from object storage every time, and fails while the store is unreachable, even for a snapshot just planned.

Per load_table plus plan_files, pyiceberg 0.12.0 (and main) reads:

  1. metadata.json on every load, on non-REST catalogs (REST returns it inline).
  2. The manifest list on every plan. _manifests re-reads it on purpose, to avoid the O(N²) memory of caching overlapping ManifestFile tuples (#1187).
  3. Each manifest on every plan, via fetch_manifest_entry. _manifest_cache keeps the ManifestFile records, not the manifests' contents.

These files are never rewritten, so the bytes at a location never change.

Iceberg Java caches these bytes. ManifestFiles and ManifestLists read through a per-FileIO ContentCache, set by these properties (CatalogProperties):

  • io.manifest.cache-enabled (default false)
  • io.manifest.cache.max-total-bytes (default 100 MiB)
  • io.manifest.cache.max-content-length (default 8 MiB)
  • io.manifest.cache.expiration-interval-ms (default 60 s)
Proposal

  1. Cache manifest and manifest-list bytes by location, off by default and bounded in bytes, as in Java. Parsing stays as it is.

    Caching bytes avoids the problem in #3978 / #3991. A ManifestFile record carries values from the manifest list that names it, so a parsed-object cache keyed by path can serve one reference's values to another. (iceberg-rust adds added_snapshot_id, sequence_number and first_row_id to its key for this reason.) Bytes are the same for every reference.

  2. Let the caller supply the cache, e.g. get(location) -> bytes | None and put(location, data), with the in-memory one as the default. A disk cache behind it would survive restarts and keep planning working through a store outage for snapshots already read. iceberg-rust has a similar hook (ObjectCacheProvide) for parsed objects.

metadata.json could use the same cache. For REST, #4075 / #4094 cover it with If-None-Match.

Java configures this with FileIO properties; #2952 used Config and an env var for manifest-cache-size. Either is fine.

Related: #4007 and #3662 make each read cheaper; this removes repeated reads.

What we do today

In Strata (a materialization layer that, among other things, caches Iceberg scans) we set py-io-impl to a FileIO that wraps the one load_file_io would pick. Its new_input serves *.metadata.json and *.avro on remote schemes from a bounded disk cache, keyed by location plus the endpoint properties, and passes everything else through.

Drawbacks:

  • It identifies immutable files by suffix.
  • It takes over py-io-impl, so it must carry a user's own py-io-impl under another property.
  • It builds an inner FileIO per scheme, since a FileIO does not know which locations it will serve.

A cache hook at the read points above would remove the need for the wrapper.

Linguagem predominante
Python
Estrelas
1.2k
Forks
654
Merge médio
2d 11h
PRs com merge (30d)
122

Preparar o ambiente

  • Sem Dockerfile nem arquivo Docker Compose
  • Tem um modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/iceberg-python

Todas as issues de apache/iceberg-python

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.