rows_from_file() splits a single-column CSV file on a letter, or raises csv.Error
Maintainers usually reply within 3 days
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 84/100
Research direction
Locate rows_from_file() and reproduce the single-column CSV cases with the sqlite-utils memory commands shown in the issue. Check how delimiter detection is performed, then run the relevant existing tests and add coverage for single-column CSV input with headers and URLs. The work is done when files without a recognized delimiter are imported as one column, while insert --sniff behavior remains unchanged.
Written by the indexing model from the issue text.
Description
When rows_from_file() detects the format of a single-column CSV file, it either fails or splits the column on a letter. That affects sqlite-utils memory file.csv and any Python call that leaves format= unset.
$ printf 'name\nAnn\nBob\n' > names.csv
$ sqlite-utils memory names.csv "select * from names"
...
_csv.Error: Could not determine delimiter
$ printf 'url\nhttps://example.com/a\nhttps://example.org/b\n' > urls.csv
$ sqlite-utils memory urls.csv "select * from urls"
[{"ur": "https://examp", "": "e.com/a"},
{"ur": "https://examp", "": "e.org/b"}]
(sqlite-utils 4.2.1)
The detection calls csv.Sniffer().sniff() on the first 2048 bytes without a delimiters= list. The sniffer then treats any character as a candidate, so a file with a single column either has no consistent candidate (csv.Error) or has a letter that happens to repeat the same number of times on every line (l above).
A fix could sniff only for the delimiters a CSV file is realistically written with (,, ;, tab, |) and read the file as one column when none of them is found. insert --sniff calls the sniffer the same way, but that is an explicit request to detect the dialect, so I would leave it unchanged.
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 215
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from simonw/sqlite-utils
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
simonw/sqlite-utils#928 ·
Maintainers usually reply within 3 days
-
Tests use CliRunner.isolated_filesystem(), which is deprecated and will be removed in Click 9.0Possibly taken @devpryan7792 claimed this 11 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
simonw/sqlite-utils#916 ·
Maintainers usually reply within 3 days
-
enable_fts() exhausts iterator columns before populating the indexPossibly taken @syahra712 claimed this 25 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
simonw/sqlite-utils#902 ·
Maintainers usually reply within 3 days
-
rows_from_file() ignores ignore_extras= and extras_key= for anything that isn't Format.CSVPossibly taken @feiiiiii5 claimed this 36 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
simonw/sqlite-utils#892 ·
Maintainers usually reply within 3 days
-
Raw numpy scalar values get silently stored as corrupted BLOBs instead of being convertedPossibly taken @TanbirRamim claimed this 43 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
simonw/sqlite-utils#876 ·
Maintainers usually reply within 3 days
All issues in simonw/sqlite-utils
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
didi/helixharness#18 ·
-
request-theme
Difficulty 1/5 Under an hour Newbie friendliness 78/100
LizardByte/ThemerrDB#9883 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
NousResearch/hermes-agent#149885 ·
Maintainers usually reply within 1 day
-
Set cache_ok = True on GUID TypeDecorator to enable SQLAlchemy query cachingPossibly taken @patelshivani2283-lab claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 91/100
fedora-infra/anitya#2157 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Project-MONAI/MONAI#9258 ·
Maintainers usually reply within 3 days