Using `sql_tokenize()` table function for tokenization.
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 45/100
Research direction
Start in src/duckdb_py/duckdb_python.cpp near PyTokenize and trace how the Python client currently tokenizes queries. Compare that path with the sql_tokenize table function introduced by duckdb/duckdb#20171; done means the client uses the table function while preserving the returned tokens and categories.
Written by the indexing model from the issue text.
Description
What happens?
Original issue here: https://github.com/duckdblabs/duckdb-internal/issues/6733
https://github.com/duckdb/duckdb/pull/20171 introduced a new table function sql_tokenize, which takes as input the query and returns the tokens and their category.
Currently in the python client we use a custom implementation somewhere near PyTokenize (see here). I think this can now be moved over to use the table function instead, making it a bit simpler, and possible to provide more information about tokenization in the future.
To Reproduce
See above
OS:
MacOS
DuckDB Package Version:
latest
Python Version:
3.14?
Full Name:
Daniel ten Wolde
Affiliation:
DuckDB Labs
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
Did you include all relevant data sets for reproducing the issue?
Yes
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration to reproduce the issue?
- Yes, I have
- Dominant language
- Python
- Stars
- 186
- Forks
- 113
- Avg merge
- 20h 58m
- Merged PRs (30d)
- 11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from duckdb/duckdb-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
duckdb/duckdb-python#627 ·
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
duckdb/duckdb-python#576 · 3 comments ·
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
duckdb/duckdb-python#534 ·
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
duckdb/duckdb-python#386 ·
-
needs triage
Difficulty 4/5 3-5 days Newbie friendliness 48/100
duckdb/duckdb-python#622 ·
All issues in duckdb/duckdb-python
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100