resolve_pattern() allowed_extensions filter accepts intermediate suffixes — c.parquet.tmp and b.csv.bak pass as parquet/CSV data files

Open Beginner friendly
#9,100 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
data

Research direction

Start in src/datasets/data_files.py around the resolve_pattern() extension filter at lines 390–396, then inspect nearby tests for allowed extensions. Reproduce with .csv and .parquet patterns and filenames such as b.csv.bak and c.parquet.tmp. Done means only files whose final extension is allowed are returned, with tests covering intermediate suffixes.

Written by the indexing model from the issue text.

Description

Found on main (39fd698d619816099ffa8eb07b5c79b0f4f5f299, datasets 5.2.1.dev0). I reproduced this live against the local checkout.

In src/datasets/data_files.py:390-396:

if any("." + suffix in allowed_extensions for suffix in xbasename(filepath).split(".")[1:])

Same dotted-component counting as the infer_module issue, but here it is a filter: any component matching is enough. So a backup/temp file such as b.csv.bak or c.parquet.tmp is accepted as a genuine .csv / .parquet data file and handed to the CSV/parquet loader, which then fails with a confusing parse error — or worse, a stale .bak copy of a dataset is silently loaded in addition to (or instead of) the real file, producing wrong data.

Repro (verified live — dir contains a.csv, b.csv.bak, c.parquet.tmp, d.txt):

from datasets.data_files import resolve_pattern
resolve_pattern("*", tmpdir, allowed_extensions=[".csv"])
# -> ['a.csv', 'b.csv.bak']   (b.csv.bak should be excluded)
resolve_pattern("*", tmpdir, allowed_extensions=[".parquet"])
# -> ['c.parquet.tmp']        (a garbage .tmp file passes as parquet)

Why it matters: the extension filter is the guard that is supposed to keep non-data files out of a load; it lets through exactly the files it exists to exclude (backups, temp files, partial downloads), turning a typo-level mess in a data directory into a wrong-data or cryptic-failure load.

I searched open issues for resolve_pattern + allowed_extensions — 2 hits (#6549, #7689), both about Hub loading error messages, unrelated.

Dominant language
Python
Stars
22.2k
Forks
3.7k
Avg merge
2d 19h
Merged PRs (30d)
19

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/datasets

All issues in huggingface/datasets

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.