Cache manifest and manifest-list bytes by location, as Iceberg Java's `io.manifest.cache-*` does

Ouverte
  • Quelque chose ne va pas ?
#4,300 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@Anurag-S1ngh y travaille déjà.

Depuis le 11/11/2026.

  • #4335 par @Anurag-S1ngh — ouverte

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
35/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
python
Domaine
databases

Piste de recherche

Start by tracing load_table, plan_files, _manifests, and fetch_manifest_entry to confirm where metadata, manifest-list, and manifest bytes are read. Then review the existing _manifest_cache and the related issues #1187, #3978, and #3991. Done means callers can supply a bounded, location-keyed byte cache for manifests and manifest lists, disabled by default, while parsing remains unchanged.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Feature Request / Improvement

Planning the same table repeatedly reads the same immutable files from object storage every time, and fails while the store is unreachable, even for a snapshot just planned.

Per load_table plus plan_files, pyiceberg 0.12.0 (and main) reads:

  1. metadata.json on every load, on non-REST catalogs (REST returns it inline).
  2. The manifest list on every plan. _manifests re-reads it on purpose, to avoid the O(N²) memory of caching overlapping ManifestFile tuples (#1187).
  3. Each manifest on every plan, via fetch_manifest_entry. _manifest_cache keeps the ManifestFile records, not the manifests' contents.

These files are never rewritten, so the bytes at a location never change.

Iceberg Java caches these bytes. ManifestFiles and ManifestLists read through a per-FileIO ContentCache, set by these properties (CatalogProperties):

  • io.manifest.cache-enabled (default false)
  • io.manifest.cache.max-total-bytes (default 100 MiB)
  • io.manifest.cache.max-content-length (default 8 MiB)
  • io.manifest.cache.expiration-interval-ms (default 60 s)
Proposal

  1. Cache manifest and manifest-list bytes by location, off by default and bounded in bytes, as in Java. Parsing stays as it is.

    Caching bytes avoids the problem in #3978 / #3991. A ManifestFile record carries values from the manifest list that names it, so a parsed-object cache keyed by path can serve one reference's values to another. (iceberg-rust adds added_snapshot_id, sequence_number and first_row_id to its key for this reason.) Bytes are the same for every reference.

  2. Let the caller supply the cache, e.g. get(location) -> bytes | None and put(location, data), with the in-memory one as the default. A disk cache behind it would survive restarts and keep planning working through a store outage for snapshots already read. iceberg-rust has a similar hook (ObjectCacheProvide) for parsed objects.

metadata.json could use the same cache. For REST, #4075 / #4094 cover it with If-None-Match.

Java configures this with FileIO properties; #2952 used Config and an env var for manifest-cache-size. Either is fine.

Related: #4007 and #3662 make each read cheaper; this removes repeated reads.

What we do today

In Strata (a materialization layer that, among other things, caches Iceberg scans) we set py-io-impl to a FileIO that wraps the one load_file_io would pick. Its new_input serves *.metadata.json and *.avro on remote schemes from a bounded disk cache, keyed by location plus the endpoint properties, and passes everything else through.

Drawbacks:

  • It identifies immutable files by suffix.
  • It takes over py-io-impl, so it must carry a user's own py-io-impl under another property.
  • It builds an inner FileIO per scheme, since a FileIO does not know which locations it will serve.

A cache hook at the read points above would remove the need for the wrapper.

Langage dominant
Python
Étoiles
1.2k
Forks
654
Merge moyen
2 j 11 h
PR mergées (30 j)
122

Préparer son environnement

  • Aucun Dockerfile ni fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de apache/iceberg-python

Toutes les issues de apache/iceberg-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.