Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

feat(python): allow a metadata cache for LanceFileReader / LanceFileSession

Open
  • Is something wrong?
#10,264 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python, rust
Domain
database, performance

Research direction

Read python/src/file.rs around the LanceFileReader::open_with_store call and the Python definitions of LanceFileSession and LanceFileReader in python/python/lance/file.py. Trace how lance.Session exposes its metadata cache and how file readers receive it; the Rust cache type and no_cache() call are identified in the issue. Done means file readers can opt into a shared session cache while the default remains uncached, with coverage for the Python API.

Written by the indexing model from the issue text.

Description

feature
Problem

Lance is increasingly used purely as a file format inside other table formats or directly. In that setting there is no Lance manifest / LanceDataset, so the only Python read entry points are LanceFileReader and LanceFileSession.open_reader().

The Python binding opens the Rust FileReader with a hard-coded zero-capacity cache: python/src/file.rs#L887 — &LanceCache::no_cache().

As a result, random access (take_rows / small read_range) re-fetches page metadata from object storage on every call (find_cache_misses), and every new reader re-reads the footer / column metadata.

There is currently no Python parameter to change this. The cache knobs (lance.Session(metadata_cache_size_bytes=..., metadata_cache_backend=...)) are only reachable through lance.dataset(), and LanceFileSession only shares the object store (#4670).

Proposal

Add an opt-in metadata cache to the Python file API. The default stays unchanged (no cache).

The preferred option reuses the existing lance.Session, so cache sizing, cache backends and observability (size_bytes()) are the same as on the dataset path:

import lance
from lance.file import LanceFileSession, LanceFileReader

session = lance.Session(metadata_cache_size_bytes=1 << 30)

# Readers opened from a file session share the session's metadata cache
fs = LanceFileSession(base_uri, storage_options=opts, session=session)
reader = fs.open_reader("data/xxx.lance")
reader.take_rows([3, 42, 7])

# Standalone readers can share the same cache
reader = LanceFileReader(uri, storage_options=opts, session=session)

session.size_bytes()  # observe cache usage
  • Alternative: a plain metadata_cache_size_bytes= argument on LanceFileSession.
  • Implementation sketch: keep the session's metadata cache (Arc<LanceCache>) in the Python LanceFileSession / LanceFileReader, and pass it instead of LanceCache::no_cache() in LanceFileReader::open_with_store. Cache keys are already prefixed by file path, so one cache can be shared safely across files.

Happy to send a PR if the direction looks acceptable.

Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
5d 1h
Merged PRs (30d)
392

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.