Hacktoberfest 2026:维护者为十月标记出来的 issue,仍然开放、适合新手。 浏览 Hacktoberfest issue

Server-side streaming cursors

未关闭
#36 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
冷清
技术栈
python, sqlalchemy

调研方向

首先阅读现有的 Cursor 和 AsyncCursor 实现,以及 SyncResponseContextIterator 和 AsyncResponseContextIterator API。使用 .github/docker/docker-compose.yml 进行本地 YDB 集成测试,并使用 fake session/stream pool 添加单元测试。当导出的同步和异步流式游标能够处理 session 所有权、事务独占、rowcount、清理以及 README 示例时,即表示完成。

由索引模型根据 Issue 内容生成。

描述

Motivation

The current Cursor / AsyncCursor implementation buffers the full
result of a query in memory before fetchone / fetchmany / fetchall
can be called. For large result sets (analytical queries, full-table
scans, ETL-style jobs) this is either infeasible or prohibitively
memory-hungry.

The YDB Query API exposes result sets as a stream
(SyncResponseContextIterator / AsyncResponseContextIterator) —
result sets arrive incrementally over the wire. A DB-API cursor that
consumes that stream lazily would let users process arbitrarily large
results with bounded memory.

Downstream use case: SQLAlchemy

SQLAlchemy has first-class support for server-side cursors via:

  • Connection.execution_options(stream_results=True)
  • Query.yield_per(N) / select(...).execution_options(yield_per=N)

For these to work, the DB-API driver must expose a cursor that fetches
from the server incrementally rather than materialising everything up
front. Without a streaming cursor on our side, SQLAlchemy users can't
use yield_per / stream_results against YDB and have to either page
manually or blow up memory.

Equivalents in other drivers:

  • psycopg2 — named (server-side) cursors: conn.cursor(name="...")
  • psycopg (v3) — conn.cursor(name="...") / ClientCursor vs
    ServerCursor
  • asyncpg — cursor objects returned from conn.cursor(query) inside
    a transaction

Proposed API

Expose a streaming variant through an extra kwarg on Connection.cursor:

with connection.cursor(stream_results=True) as cur:
    cur.execute("SELECT ... FROM huge_table")
    for row in iter(cur.fetchone, None):
        ...

And for async:

async with async_connection.cursor(stream_results=True) as cur:
    await cur.execute("SELECT ... FROM huge_table")
    while (row := await cur.fetchone()) is not None:
        ...

New public classes: StreamCursor, AsyncStreamCursor, exported from
ydb_dbapi.

Scope

  • StreamCursor (sync) consuming SyncResponseContextIterator
  • AsyncStreamCursor consuming AsyncResponseContextIterator
  • Own-session mode (auto-commit, no interactive tx): cursor acquires
    a session from the pool for the duration of the stream and
    releases on finish / close / error
  • Interactive-tx mode: stream runs on the connection's tx session,
    with exclusivity — while a stream is active no other cursor on
    the same connection may execute (would corrupt the tx session);
    commit / rollback should reject while a stream is running
  • rowcount semantics for streaming (likely -1 until drained)
  • close() must cleanly terminate a mid-flight stream (cancel +
    discard session, or drain) in both sync and async paths
  • Integration tests against a local YDB (see
    .github/docker/docker-compose.yml)
  • Unit tests with fake session/stream pools
  • README documentation for the new flag + both code examples
  • SQLAlchemy dialect wiring for stream_results /
    yield_per — likely a follow-up in ydb-sqlalchemy, but worth
    mentioning here
主要语言
Python
星标
4
派生
2
平均合并
6 天 19 小时
30 天内合并 PR
1

贡献指南

这个仓库没有索引到贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。