Translate unsupported fsspec modification times into a typed DuckDB error
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 68/100
Research direction
Start at PythonFilesystem::GetLastModifiedTime() and trace how DirectFileReader handles optional metadata errors. Add a regression test using a custom fsspec filesystem with unsupported modified() behavior, then run it through read_blob(...).last_modified. Done means unsupported modification times return NULL while unrelated filesystem errors still surface.
Written by the indexing model from the issue text.
Description
What happens?
When a filesystem registered through Connection.register_filesystem() cannot provide a modification time, querying the last_modified column of read_blob surfaces the raw Python exception instead of returning NULL.
One concrete case is gcsfs: GCSFileSystem.modified() indexes info(path)["mtime"], while some GCS object metadata (and synthesized directory entries) has no mtime. The resulting KeyError: 'mtime' currently reaches a DuckDB query as an untyped error and can abort callers such as DuckLake CHECKPOINT.
This is the Python/fsspec half of:
- duckdb/ducklake#1042
- duckdb/ducklake#1336
Why the error loses its type
PythonFilesystem::GetLastModifiedTime() currently calls filesystem.modified() directly. A Python exception crosses the nanobind boundary as std::exception; when DuckDB later constructs ErrorData(ex), it has ExceptionType::INVALID, so extensions cannot distinguish unsupported modification-time metadata from unrelated I/O failures without inspecting the message.
DuckDB core already has the desired downstream behavior: DirectFileReader converts ExceptionType::NOT_IMPLEMENTED failures for optional file metadata columns to NULL. An older_than predicate then naturally retains only files whose age is unknown.
Suggested direction
Translate the Python filesystem capability error at the adapter boundary in PythonFilesystem::GetLastModifiedTime():
- Catch
nb::python_error. - Map Python
NotImplementedErrorto DuckDBNotImplementedException. - Consider a narrowly scoped compatibility mapping for the current gcsfs
KeyError('mtime'), or coordinate with gcsfs so missingmtimeis reported asNotImplementedError. - Re-throw every unrelated Python exception unchanged.
Please add a regression test with a small custom fsspec filesystem whose modified() is unsupported, asserting that read_blob(...).last_modified is NULL while other filesystem errors still surface.
This keeps provider-specific exception classification in the Python filesystem adapter instead of requiring downstream extensions to accumulate message-matching helpers.
- Dominant language
- Python
- Stars
- 186
- Forks
- 113
- Avg merge
- 20h 58m
- Merged PRs (30d)
- 11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from duckdb/duckdb-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
duckdb/duckdb-python#627 ·
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
duckdb/duckdb-python#576 · 3 comments ·
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
duckdb/duckdb-python#534 ·
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
duckdb/duckdb-python#386 ·
-
needs triage
Difficulty 4/5 3-5 days Newbie friendliness 48/100
duckdb/duckdb-python#622 ·
All issues in duckdb/duckdb-python
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100