`CachingFileManager` opens one file handle per `open_dataset()` call — could identical managers share a `FILE_CACHE` entry?

Open
#11,581 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
python
Domain
performance

Research direction

Start with CachingFileManager._make_key(), FILE_CACHE, and the open_dataset() entry point described in the issue. Determine whether identical live managers can safely share a cache entry while preserving the behavior addressed by #4879; done means a concrete, validated sharing or cache-accounting design.

Written by the indexing model from the issue text.

Description

enhancement needs discussion topic-backends
What is your issue?

Disclaimer: My understanding of Xarray internals provided by Claude.

I have a situation in my downstream library Satpy where code was written to open a NetCDF file with open_dataset multiple times. The files I was dealing with had a lot of groups and loading a dataset included ~40 files. The initial naive implementation called open_dataset once per variable and loaded things as dask arrays. Due to a bug in HDF5 which I've filed privately upstream this was causing a segmentation fault when xarray was closing one cached entry even though others for the same file still existed and were open.

CachingFileManager._make_key() includes self._manager_id, a fresh uuid4() per
manager. So N open_dataset() calls on the same path with identical kwargs produce N
independent handles, each taking its own FILE_CACHE slot. With the default
file_cache_maxsize=128 and our naive implementation we end up thrashing the LRU cache even for a single file. Since every open_dataset is a new entry that means the file is reopened and metadata is reparsed (at least that's what Claude says). Under the hood my understanding is that HDF5 is only keeping one actual open file in modern HDF5 versions.

I'm wondering whether that is intended in this case, or whether managers that are
simultaneously alive and compare equal on (opener, args, mode, kwargs) could share one
open file? I know there was #4879 to fix #4240 and #4862. I'm not saying that I disagree with that use case and those changes, but wondering if a further optimization can be done (even if opt-in) to share a single Dataset for the same combination. I think Claude said that even for group-specific open_dataset calls the root/top-level NetCDF4.Dataset object is held onto? So maybe sharing the entry for all groups would work too.

Claude suggested two possible directions that I'll summarize here:

  1. To keep the #4879 fix working and prevent new open_dataset calls not reusing an old cache entry, make the caching smarter to see "does this cache entry have any active managers", reuse that entry if so or open a new version if no active managers.
  2. Make the LRU cache more understanding of entries that share the same paths and handle them differently. Like sizing/accounting per path. I don't see how this would be possible as the cache understanding that two or more entries are "equivalent" would probably mean they should just use the same entry in the first place.
Dominant language
Python
Stars
4.2k
Forks
1.5k
Avg merge
2d 21h
Merged PRs (30d)
74

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pydata/xarray

All issues in pydata/xarray

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.