Server-side streaming cursors
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python, sqlalchemy
- Domain
- backend-api-design, database
Research direction
Start by reading the existing Cursor and AsyncCursor implementations and the SyncResponseContextIterator and AsyncResponseContextIterator APIs. Use .github/docker/docker-compose.yml for local YDB integration tests, and add unit tests with fake session/stream pools. Done means exported sync and async streaming cursors handle session ownership, transaction exclusivity, rowcount, cleanup, and README examples.
Written by the indexing model from the issue text.
Description
Motivation
The current Cursor / AsyncCursor implementation buffers the full
result of a query in memory before fetchone / fetchmany / fetchall
can be called. For large result sets (analytical queries, full-table
scans, ETL-style jobs) this is either infeasible or prohibitively
memory-hungry.
The YDB Query API exposes result sets as a stream
(SyncResponseContextIterator / AsyncResponseContextIterator) —
result sets arrive incrementally over the wire. A DB-API cursor that
consumes that stream lazily would let users process arbitrarily large
results with bounded memory.
Downstream use case: SQLAlchemy
SQLAlchemy has first-class support for server-side cursors via:
Connection.execution_options(stream_results=True)Query.yield_per(N)/select(...).execution_options(yield_per=N)
For these to work, the DB-API driver must expose a cursor that fetches
from the server incrementally rather than materialising everything up
front. Without a streaming cursor on our side, SQLAlchemy users can't
use yield_per / stream_results against YDB and have to either page
manually or blow up memory.
Equivalents in other drivers:
psycopg2— named (server-side) cursors:conn.cursor(name="...")psycopg(v3) —conn.cursor(name="...")/ClientCursorvs
ServerCursorasyncpg— cursor objects returned fromconn.cursor(query)inside
a transaction
Proposed API
Expose a streaming variant through an extra kwarg on Connection.cursor:
with connection.cursor(stream_results=True) as cur:
cur.execute("SELECT ... FROM huge_table")
for row in iter(cur.fetchone, None):
...
And for async:
async with async_connection.cursor(stream_results=True) as cur:
await cur.execute("SELECT ... FROM huge_table")
while (row := await cur.fetchone()) is not None:
...
New public classes: StreamCursor, AsyncStreamCursor, exported from
ydb_dbapi.
Scope
-
StreamCursor(sync) consumingSyncResponseContextIterator -
AsyncStreamCursorconsumingAsyncResponseContextIterator - Own-session mode (auto-commit, no interactive tx): cursor acquires
a session from the pool for the duration of the stream and
releases on finish / close / error - Interactive-tx mode: stream runs on the connection's tx session,
with exclusivity — while a stream is active no other cursor on
the same connection may execute (would corrupt the tx session);
commit/rollbackshould reject while a stream is running -
rowcountsemantics for streaming (likely-1until drained) -
close()must cleanly terminate a mid-flight stream (cancel +
discard session, or drain) in both sync and async paths - Integration tests against a local YDB (see
.github/docker/docker-compose.yml) - Unit tests with fake session/stream pools
- README documentation for the new flag + both code examples
- SQLAlchemy dialect wiring for
stream_results/
yield_per— likely a follow-up inydb-sqlalchemy, but worth
mentioning here
- Dominant language
- Python
- Stars
- 4
- Forks
- 2
- Avg merge
- 6d 19h
- Merged PRs (30d)
- 1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Similar issues
-
bug confirmed issue
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
open-webui/open-webui#30750 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
OpenwaterHealth/openmotion-bloodflow-app#604 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
good first issue
Difficulty 1/5 Under an hour Newbie friendliness 90/100