DirCache.items() and values() raise RuntimeError with bounded LRU caching

Open Beginner friendly
  • Is something wrong?
#2,419 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Junaid-006 is already working on this.

Since Nov 2, 2026.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
65/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend

Research direction

Start in fsspec/dircache.py with DirCache.iter, which yields from the internal OrderedDict when no expiry is set, and getitem, whose LRU update reorders that same dict. Run the reproduction from the issue with max_paths=2 to confirm the RuntimeError. Done means list(cache.items()) and list(cache.values()) succeed under bounded LRU, with existing dircache tests still passing. Check first whether the overlap with the open redesign in PR #2094 affects scope.

Written by the indexing model from the issue text.

Description

Summary

On current master (b13312fc1f3634e2f3c5cd5dcaf812a11d03bdfe), materializing the standard mapping views from DirCache raises RuntimeError: OrderedDict mutated during iteration when max_paths is positive, at least two entries are retained, and listings_expiry_time is left at its default None.

Reproduction

from fsspec.dircache import DirCache

cache = DirCache(max_paths=2)
cache["/a"] = [{"name": "/a/one", "type": "file", "size": 1}]
cache["/b"] = [{"name": "/b/two", "type": "file", "size": 2}]

list(cache.items())
# RuntimeError: OrderedDict mutated during iteration

A fresh cache with the same entries also fails on list(cache.values()). list(cache.keys()) succeeds. The same items() and values() operations succeed with unlimited capacity, only one retained entry, or an unexpired listings_expiry_time=3600.

Expected behavior and cause

Iterating the inherited MutableMapping item/value views should return the cached directory listings without raising. No application code inserts or deletes entries while iterating.

Without expiry, DirCache.__iter__ yields directly from _cache. The item/value views then call __getitem__, whose LRU update moves the key in that same OrderedDict, invalidating its active iterator. With expiry configured, __iter__ already uses a key snapshot, so the same operation succeeds.

The observed effect is failed cache-content enumeration. I have not demonstrated missing file data or a failure in every filesystem backend.

Related work and scope question

I checked all 110 currently open PRs and their 436 changed-file entries, read the relevant dircache.py changes in #895, #1396, #2094 and #2398, and read the earlier #2095 / #2097 / #2165 discussions. I did not find an existing targeted report of this precise failure in the searches performed; this cannot exclude unpublished work.

There is overlap with the broad metadata-cache redesign in #2094: its snapshot iterator avoids this precise exception in a source-only check, although it also changes cache behavior and has an outstanding review finding. I do not want to duplicate or compete with that work. Would a small, separate fix retaining the current DirCache contract be useful while that redesign remains open, or would you prefer this handled there?

Verification and disclosure

Observed on Windows, Python 3.13.15, using the unchanged fsspec/dircache.py file fetched from the master SHA above. Its Git blob matches the pinned tree. The module was loaded directly with Python's import machinery; this is not a full-package or backend-integration run. A 40-case probe produced four expected reproductions of this exception and 36 successful controls; it did not modify DirCache or run the repository's full suite.

This report was identified and prepared with Hermes Agent using OpenAI GPT-6.1. The examples above are minimal synthetic reproductions, not a claim that I encountered the failure in production.

Dominant language
Python
Stars
1.4k
Forks
570
Avg merge
2d 18h
Merged PRs (30d)
102

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from fsspec/filesystem_spec

All issues in fsspec/filesystem_spec

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.