Server-side streaming cursors
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 冷清
- 技术栈
- python, sqlalchemy
调研方向
首先阅读现有的 Cursor 和 AsyncCursor 实现,以及 SyncResponseContextIterator 和 AsyncResponseContextIterator API。使用 .github/docker/docker-compose.yml 进行本地 YDB 集成测试,并使用 fake session/stream pool 添加单元测试。当导出的同步和异步流式游标能够处理 session 所有权、事务独占、rowcount、清理以及 README 示例时,即表示完成。
由索引模型根据 Issue 内容生成。
描述
Motivation
The current Cursor / AsyncCursor implementation buffers the full
result of a query in memory before fetchone / fetchmany / fetchall
can be called. For large result sets (analytical queries, full-table
scans, ETL-style jobs) this is either infeasible or prohibitively
memory-hungry.
The YDB Query API exposes result sets as a stream
(SyncResponseContextIterator / AsyncResponseContextIterator) —
result sets arrive incrementally over the wire. A DB-API cursor that
consumes that stream lazily would let users process arbitrarily large
results with bounded memory.
Downstream use case: SQLAlchemy
SQLAlchemy has first-class support for server-side cursors via:
Connection.execution_options(stream_results=True)Query.yield_per(N)/select(...).execution_options(yield_per=N)
For these to work, the DB-API driver must expose a cursor that fetches
from the server incrementally rather than materialising everything up
front. Without a streaming cursor on our side, SQLAlchemy users can't
use yield_per / stream_results against YDB and have to either page
manually or blow up memory.
Equivalents in other drivers:
psycopg2— named (server-side) cursors:conn.cursor(name="...")psycopg(v3) —conn.cursor(name="...")/ClientCursorvs
ServerCursorasyncpg— cursor objects returned fromconn.cursor(query)inside
a transaction
Proposed API
Expose a streaming variant through an extra kwarg on Connection.cursor:
with connection.cursor(stream_results=True) as cur:
cur.execute("SELECT ... FROM huge_table")
for row in iter(cur.fetchone, None):
...
And for async:
async with async_connection.cursor(stream_results=True) as cur:
await cur.execute("SELECT ... FROM huge_table")
while (row := await cur.fetchone()) is not None:
...
New public classes: StreamCursor, AsyncStreamCursor, exported from
ydb_dbapi.
Scope
-
StreamCursor(sync) consumingSyncResponseContextIterator -
AsyncStreamCursorconsumingAsyncResponseContextIterator - Own-session mode (auto-commit, no interactive tx): cursor acquires
a session from the pool for the duration of the stream and
releases on finish / close / error - Interactive-tx mode: stream runs on the connection's tx session,
with exclusivity — while a stream is active no other cursor on
the same connection may execute (would corrupt the tx session);
commit/rollbackshould reject while a stream is running -
rowcountsemantics for streaming (likely-1until drained) -
close()must cleanly terminate a mid-flight stream (cancel +
discard session, or drain) in both sync and async paths - Integration tests against a local YDB (see
.github/docker/docker-compose.yml) - Unit tests with fake session/stream pools
- README documentation for the new flag + both code examples
- SQLAlchemy dialect wiring for
stream_results/
yield_per— likely a follow-up inydb-sqlalchemy, but worth
mentioning here
- 主要语言
- Python
- 星标
- 4
- 派生
- 2
- 平均合并
- 6 天 19 小时
- 30 天内合并 PR
- 1
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
相似的 Issue
-
area: harness bug status: needs-triage
难度 2/5 1-3 小时 新手友好度 75/100
Human-Agent-Society/reef#625 ·
-
难度 2/5 1-3 小时 新手友好度 70/100
-
难度 1/5 1 小时以内 新手友好度 80/100
learningequality/kolibri#15351 · 2 条评论 ·
-
难度 2/5 1-3 小时 新手友好度 75/100
-
Name consistency 未关闭
难度 2/5 1-3 小时 新手友好度 75/100
eellak/triplestore#65 · 1 条评论 ·