Idea: arrow_record_batches cursor method
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Start by locating the cursor implementation of fetchmany_arrow and any existing iterator methods, then inspect how returned Arrow tables expose their record batches. Confirm the intended batching and iteration semantics before proposing an API; done should include a documented cursor method with tests covering iteration and the requested batch-size behavior.
Written by the indexing model from the issue text.
Description
When you call fetchmany_arrow(batchsize) and specify a batch size, you get a table that has multiple record batches.
In my experience, the record batches are much smaller than the batch size I specify. I think the SQL connector has to do record keeping to align the batch size I give and the record batches it gets from the server (IIUC).
When I call fetchmany_arrow, I end up with nested loops. The outer loops loops over fetchmany_arrow calls and the inner loops over the batches returned.
I suspect it would be less bother for everyone if there was an API (e.g. arrow_record_batches()) that returned a record-batch iterator.
- Dominant language
- Python
- Stars
- 233
- Forks
- 152
- Avg merge
- 21h 5m
- Merged PRs (30d)
- 10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from databricks/databricks-sql-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
All issues in databricks/databricks-sql-python
Similar issues
-
essnmx good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
[Feature] 奇物选择添加优先级 Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
syfoud/Simulated_Scepter#174 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Giskard-AI/giskard-oss#2840 · 1 comment ·
-
A claim comment carrying the issue number is silently declined while the workflow reports success Openarea: repo bug perceived difficulty: 2
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
yeti-platform/yeti#1380 ·