Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Process-global manifest cache is keyed only on manifest_path

Open
#3,978 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data-engineering, databases

Research direction

Start in pyiceberg/manifest.py by reading _ManifestCache.get_or_cache and the read_manifest_list entry point. Reproduce the collision with two manifest lists that use the same manifest_path but provide different metadata. Done means cache behavior no longer allows one manifest list's ManifestFile metadata to affect another table's scan pruning, with regression coverage for the collision.

Written by the indexing model from the issue text.

Description

bug

pyiceberg/manifest.py holds a module-level cache:

_manifest_cache = _ManifestCache()

Its docstring describes it accurately: "Process-wide ManifestFile cache keyed by manifest_path." The key is the path string alone — no catalog, table, session, or tenant forms part of it.

def get_or_cache(self, manifest_file: ManifestFile) -> ManifestFile:
    ...
    manifest_path = manifest_file.manifest_path
    if manifest_path in self._cache:
        return self._cache[manifest_path]
    self._cache[manifest_path] = manifest_file
    return manifest_file

On a hit it returns the cached object and discards the one just read. Since read_manifest_list constructs ManifestFile objects from the entries of a manifest list rather than by reading each manifest file, the cached object reflects whatever the first manifest list to name that path declared — including its partition summaries, counts and sequence numbers.

Two tables in the same process that reference the same manifest_path therefore share one ManifestFile, and the first read wins. Because those fields drive scan pruning, a stale or mismatched entry changes which files a scan considers.

Issue investigation generated via claude, reviewed by Sung, Kevin, Fokko.

Dominant language
Python
Stars
1.1k
Forks
589
Avg merge
2d 4h
Merged PRs (30d)
72

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg-python

All issues in apache/iceberg-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.