duckdb sniff_csv fails with > 10240 lines
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 38/100
Research direction
Start with the provided Python execute call using read_csv, header=True, delim=';', and sample_size values of 10240 and 10241. Investigate why sniff_csv fails when sampling beyond 10240 lines; done means the reproduction no longer raises an "unable to detect csv format" error for larger sample sizes.
Written by the indexing model from the issue text.
Description
What happens?
When I use sniff_csv implicitly using read_csv with the arguments ", header=True, delim=';', sample_size=10241", it triggers an "unable to detect csv format"-error.
- I can place multiple different lines on line 20241 and they all trigger it. The lines seem fine and not different than any others.
- When I reduce the sample_size to 10240, the import works again.
- When I then remove lines 10236-10240 from the original file and keep the sample_size on 10240, it also works.
This indicates to me that 10240 is an upper limit for the sampling before it goes wrong, but the default is of 20xxx.
I unfortunately can't provide you the sample.
The duckdb-version is "v1.2.0"
To Reproduce
self._db.execute(f"CREATE OR REPLACE TABLE {all_table_name} AS SELECT * FROM read_csv('{self.event.tmp_file_path}', header=True, delim=';', sample_size=10241)")
OS:
Linux
DuckDB Package Version:
1.2.0
Python Version:
3.13.7
Full Name:
Michel
Affiliation:
Acme
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
Did you include all relevant data sets for reproducing the issue?
No - I cannot share the data sets because they are confidential
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration to reproduce the issue?
- Yes, I have
- Dominant language
- Python
- Stars
- 189
- Forks
- 117
- Avg merge
- 1d 58m
- Merged PRs (30d)
- 15
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from duckdb/duckdb-python
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
duckdb/duckdb-python#633 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
duckdb/duckdb-python#627 · 1 comment ·
Maintainers usually reply within 1 day
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
duckdb/duckdb-python#576 · 3 comments ·
Maintainers usually reply within 1 day
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
duckdb/duckdb-python#534 ·
Maintainers usually reply within 1 day
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
duckdb/duckdb-python#386 ·
Maintainers usually reply within 1 day
All issues in duckdb/duckdb-python
Similar issues
-
New InternshipOpennew_internship
Difficulty 1/5 Under an hour Newbie friendliness 70/100
-
[BUG] Reports tab: "Unban" button tooltip shows raw `{{ip}}` placeholder instead of the IP addressOpenbug javascript ui
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
bunkerity/bunkerweb#4001 · 1 comment ·
Maintainers usually reply within 1 day
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 92/100
PedestrianDynamics/pyFDS-Evac#476 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
google/differential-privacy#516 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
adobe-fonts/source-serif#153 ·