Idea: arrow_record_batches cursor method

Open

#375 opened on Mar 20, 2024 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 5/5
Estimated time: Over a week
Newbie friendliness: 35/100

Issue type: Feature
Clarity: Mostly clear
Activity status: Stale
Tech stack: python
Domain: api

Research direction

Start by locating the cursor implementation of fetchmany_arrow and any existing iterator methods, then inspect how returned Arrow tables expose their record batches. Confirm the intended batching and iteration semantics before proposing an API; done should include a documented cursor method with tests covering iteration and the requested batch-size behavior.

Written by the indexing model from the issue text.

Description

enhancement

When you call fetchmany_arrow(batchsize) and specify a batch size, you get a table that has multiple record batches.

In my experience, the record batches are much smaller than the batch size I specify. I think the SQL connector has to do record keeping to align the batch size I give and the record batches it gets from the server (IIUC).

When I call fetchmany_arrow, I end up with nested loops. The outer loops loops over fetchmany_arrow calls and the inner loops over the batches returned.

I suspect it would be less bother for everyone if there was an API (e.g. arrow_record_batches()) that returned a record-batch iterator.

Dominant language: Python
Stars: 233
Forks: 152
Avg merge: 21h 5m
Merged PRs (30d): 10

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

slack-gif-creator: validate_gif() multiplies the last frame's duration by the frame count, so reported duration/fps are wrong (1.0s GIF reported as 1.8s) Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

anthropics/skills#1811 · 1 comment · Opened Sep 23, 2026
App fails to start with gradio 6 (allowed by `gradio>=5.49.1`) Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

speaches-ai/speaches#678 · Opened Sep 23, 2026
ModuleNotFoundError: No module named 'mcp.server.fastmcp'. Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

datalayer/mcp-compose#42 · Opened Sep 23, 2026
import spacy fails with ModuleNotFoundError: No module named 'click' Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

conda-forge/spacy-feedstock#177 · Opened Sep 23, 2026
humaneval: extract_function_body corrupts a fenced body-only reply, so the same correct code scores INCORRECT fenced and CORRECT unfenced Open

Difficulty 2/5 1-3 hours Newbie friendliness 70/100

UKGovernmentBEIS/inspect_evals#2523 · Opened Sep 23, 2026

More Python issues