resolve_pattern() allowed_extensions filter accepts intermediate suffixes — c.parquet.tmp and b.csv.bak pass as parquet/CSV data files
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Start in src/datasets/data_files.py around the resolve_pattern() extension filter at lines 390–396, then inspect nearby tests for allowed extensions. Reproduce with .csv and .parquet patterns and filenames such as b.csv.bak and c.parquet.tmp. Done means only files whose final extension is allowed are returned, with tests covering intermediate suffixes.
Written by the indexing model from the issue text.
Description
Found on main (39fd698d619816099ffa8eb07b5c79b0f4f5f299, datasets 5.2.1.dev0). I reproduced this live against the local checkout.
In src/datasets/data_files.py:390-396:
if any("." + suffix in allowed_extensions for suffix in xbasename(filepath).split(".")[1:])
Same dotted-component counting as the infer_module issue, but here it is a filter: any component matching is enough. So a backup/temp file such as b.csv.bak or c.parquet.tmp is accepted as a genuine .csv / .parquet data file and handed to the CSV/parquet loader, which then fails with a confusing parse error — or worse, a stale .bak copy of a dataset is silently loaded in addition to (or instead of) the real file, producing wrong data.
Repro (verified live — dir contains a.csv, b.csv.bak, c.parquet.tmp, d.txt):
from datasets.data_files import resolve_pattern
resolve_pattern("*", tmpdir, allowed_extensions=[".csv"])
# -> ['a.csv', 'b.csv.bak'] (b.csv.bak should be excluded)
resolve_pattern("*", tmpdir, allowed_extensions=[".parquet"])
# -> ['c.parquet.tmp'] (a garbage .tmp file passes as parquet)
Why it matters: the extension filter is the guard that is supposed to keep non-data files out of a load; it lets through exactly the files it exists to exclude (backups, temp files, partial downloads), turning a typo-level mess in a data directory into a wrong-data or cryptic-failure load.
I searched open issues for resolve_pattern + allowed_extensions — 2 hits (#6549, #7689), both about Hub loading error messages, unrelated.
- Dominant language
- Python
- Stars
- 22.2k
- Forks
- 3.7k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 19
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/datasets
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
huggingface/datasets#9099 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
huggingface/datasets#9098 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
huggingface/datasets#9097 ·
Maintainers usually reply within 1 day
-
`eval` loader fails when the first log in a folder has no samplesPossibly taken @BahaaMattar claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
huggingface/datasets#9069 ·
Maintainers usually reply within 1 day
-
JSON loader rounds floats to 10 decimals in batches that contain a mixed-type fieldPossibly taken @uvcode01-max claimed this 6 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
huggingface/datasets#9043 ·
Maintainers usually reply within 1 day
All issues in huggingface/datasets
Similar issues
-
[Bug]: reme CLI crashes when the running server used a relative config pathPossibly taken @nurlanmnn claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
agentscope-ai/ReMe#770 ·
Maintainers usually reply within 1 day
-
Pi: paused model calls are reported as "not signed in" when the model's provider comes from an extensionPossibly taken @balcsida claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
microsoft/TokenEconomizer#288 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 67/100
sickn33/agentic-awesome-skills#2437 ·
Maintainers usually reply within 1 day
-
site-bug triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ethereum/consensus-specs#5846 ·
Maintainers usually reply within 1 day