Cache manifest and manifest-list bytes by location, as Iceberg Java's `io.manifest.cache-*` does

Aperta
  • Qualcosa non va?
#4,300 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Anurag-S1ngh ci sta già lavorando.

Dal 11/11/2026.

  • #4335 di @Anurag-S1ngh — aperta

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
databases

Direzione di ricerca

Start by tracing load_table, plan_files, _manifests, and fetch_manifest_entry to confirm where metadata, manifest-list, and manifest bytes are read. Then review the existing _manifest_cache and the related issues #1187, #3978, and #3991. Done means callers can supply a bounded, location-keyed byte cache for manifests and manifest lists, disabled by default, while parsing remains unchanged.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Feature Request / Improvement

Planning the same table repeatedly reads the same immutable files from object storage every time, and fails while the store is unreachable, even for a snapshot just planned.

Per load_table plus plan_files, pyiceberg 0.12.0 (and main) reads:

  1. metadata.json on every load, on non-REST catalogs (REST returns it inline).
  2. The manifest list on every plan. _manifests re-reads it on purpose, to avoid the O(N²) memory of caching overlapping ManifestFile tuples (#1187).
  3. Each manifest on every plan, via fetch_manifest_entry. _manifest_cache keeps the ManifestFile records, not the manifests' contents.

These files are never rewritten, so the bytes at a location never change.

Iceberg Java caches these bytes. ManifestFiles and ManifestLists read through a per-FileIO ContentCache, set by these properties (CatalogProperties):

  • io.manifest.cache-enabled (default false)
  • io.manifest.cache.max-total-bytes (default 100 MiB)
  • io.manifest.cache.max-content-length (default 8 MiB)
  • io.manifest.cache.expiration-interval-ms (default 60 s)
Proposal

  1. Cache manifest and manifest-list bytes by location, off by default and bounded in bytes, as in Java. Parsing stays as it is.

    Caching bytes avoids the problem in #3978 / #3991. A ManifestFile record carries values from the manifest list that names it, so a parsed-object cache keyed by path can serve one reference's values to another. (iceberg-rust adds added_snapshot_id, sequence_number and first_row_id to its key for this reason.) Bytes are the same for every reference.

  2. Let the caller supply the cache, e.g. get(location) -> bytes | None and put(location, data), with the in-memory one as the default. A disk cache behind it would survive restarts and keep planning working through a store outage for snapshots already read. iceberg-rust has a similar hook (ObjectCacheProvide) for parsed objects.

metadata.json could use the same cache. For REST, #4075 / #4094 cover it with If-None-Match.

Java configures this with FileIO properties; #2952 used Config and an env var for manifest-cache-size. Either is fine.

Related: #4007 and #3662 make each read cheaper; this removes repeated reads.

What we do today

In Strata (a materialization layer that, among other things, caches Iceberg scans) we set py-io-impl to a FileIO that wraps the one load_file_io would pick. Its new_input serves *.metadata.json and *.avro on remote schemes from a bounded disk cache, keyed by location plus the endpoint properties, and passes everything else through.

Drawbacks:

  • It identifies immutable files by suffix.
  • It takes over py-io-impl, so it must carry a user's own py-io-impl under another property.
  • It builds an inner FileIO per scheme, since a FileIO does not know which locations it will serve.

A cache hook at the read points above would remove the need for the wrapper.

Lingua principale
Python
Stelle
1.2k
Fork
654
Merge medio
2g 11h
PR unite (30g)
122

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/iceberg-python

Tutte le issue di apache/iceberg-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.