Cache manifest and manifest-list bytes by location, as Iceberg Java's `io.manifest.cache-*` does

未关闭
  • 哪里不对？
#4,300 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@Anurag-S1ngh 已经在做这个了。

开始于 2026年11月11日。

  • #4335 来自 @Anurag-S1ngh —— 未关闭

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
python
领域
databases

调研方向

Start by tracing load_table, plan_files, _manifests, and fetch_manifest_entry to confirm where metadata, manifest-list, and manifest bytes are read. Then review the existing _manifest_cache and the related issues #1187, #3978, and #3991. Done means callers can supply a bounded, location-keyed byte cache for manifests and manifest lists, disabled by default, while parsing remains unchanged.

由索引模型根据 Issue 内容生成。

描述

Feature Request / Improvement

Planning the same table repeatedly reads the same immutable files from object storage every time, and fails while the store is unreachable, even for a snapshot just planned.

Per load_table plus plan_files, pyiceberg 0.12.0 (and main) reads:

  1. metadata.json on every load, on non-REST catalogs (REST returns it inline).
  2. The manifest list on every plan. _manifests re-reads it on purpose, to avoid the O(N²) memory of caching overlapping ManifestFile tuples (#1187).
  3. Each manifest on every plan, via fetch_manifest_entry. _manifest_cache keeps the ManifestFile records, not the manifests' contents.

These files are never rewritten, so the bytes at a location never change.

Iceberg Java caches these bytes. ManifestFiles and ManifestLists read through a per-FileIO ContentCache, set by these properties (CatalogProperties):

  • io.manifest.cache-enabled (default false)
  • io.manifest.cache.max-total-bytes (default 100 MiB)
  • io.manifest.cache.max-content-length (default 8 MiB)
  • io.manifest.cache.expiration-interval-ms (default 60 s)
Proposal

  1. Cache manifest and manifest-list bytes by location, off by default and bounded in bytes, as in Java. Parsing stays as it is.

    Caching bytes avoids the problem in #3978 / #3991. A ManifestFile record carries values from the manifest list that names it, so a parsed-object cache keyed by path can serve one reference's values to another. (iceberg-rust adds added_snapshot_id, sequence_number and first_row_id to its key for this reason.) Bytes are the same for every reference.

  2. Let the caller supply the cache, e.g. get(location) -> bytes | None and put(location, data), with the in-memory one as the default. A disk cache behind it would survive restarts and keep planning working through a store outage for snapshots already read. iceberg-rust has a similar hook (ObjectCacheProvide) for parsed objects.

metadata.json could use the same cache. For REST, #4075 / #4094 cover it with If-None-Match.

Java configures this with FileIO properties; #2952 used Config and an env var for manifest-cache-size. Either is fine.

Related: #4007 and #3662 make each read cheaper; this removes repeated reads.

What we do today

In Strata (a materialization layer that, among other things, caches Iceberg scans) we set py-io-impl to a FileIO that wraps the one load_file_io would pick. Its new_input serves *.metadata.json and *.avro on remote schemes from a bounded disk cache, keyed by location plus the endpoint properties, and passes everything else through.

Drawbacks:

  • It identifies immutable files by suffix.
  • It takes over py-io-impl, so it must carry a user's own py-io-impl under another property.
  • It builds an inner FileIO per scheme, since a FileIO does not know which locations it will serve.

A cache hook at the read points above would remove the need for the wrapper.

主要语言
Python
星标
1.2k
派生
654
平均合并
2 天 11 小时
30 天内合并 PR
122

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/iceberg-python 的其他 Issue

查看 apache/iceberg-python 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。