Hacktoberfest 2026：維護者為十月標記出來的 issue，仍然開放、適合新手。 瀏覽 Hacktoberfest issue

DataFusion plans a CTE once per reference, so ML queries grow exponentially with depth

未關閉 適合新手
  • 哪裡不對？
#267 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

還沒有人認領這個 Issue。

評估

難度
2/5
預估耗時
1-3 小時
新手友好度
64/100
Issue 類型
文件
描述清晰度
基本清楚
活躍度
活躍
技術堆疊
postgresql, python
領域
backend, data-engineering

研究方向

從 docs/limitations.md 開始，issue 將其指定為記錄 DataFusion 對 CTE 的重複規劃，以及將中間結果註冊為資料表這項替代方法的文件位置。將文件記載的行為與回報的 DataFusion 54 限制進行核對，並說明對實體化 CTE 的支援和自動決策屬於 upstream 工作。讀者能夠理解該限制並找出替代方法時，即為完成。

由索引模型根據 Issue 內容生成。

描述

enhancement

DataFusion plans a CTE once for every place that reads it, so a query whose CTEs read earlier CTEs more than once grows exponentially with its depth. DuckDB computes such a CTE once. ML models written in SQL hit this constantly: a layer norm reads its input three times, a residual connection twice, and a message-passing round reads the node states as sender, receiver and residual.

Reproduction

Each CTE joins the previous one with itself:

from datafusion import SessionContext

def query(depth):
    ctes = ["c0 AS (SELECT i, x FROM t)"] + [
        f"c{d} AS (SELECT a.i, a.x + b.x AS x FROM c{d-1} a JOIN c{d-1} b ON a.i = b.i)"
        for d in range(1, depth + 1)
    ]
    return f"WITH {', '.join(ctes)} SELECT SUM(x) FROM c{depth}"

ctx = SessionContext()
ctx.sql("CREATE TABLE t AS SELECT * FROM (VALUES (1, 1.0), (2, 2.0)) v(i, x)").collect()
plan = ctx.sql(query(8)).execution_plan().display_indent()
depth DataFusion operators DataFusion table scans DuckDB 1.5.5 operators
2 16 4 13
4 64 16 23
6 256 64 33
8 1,024 256 43

In the ML benchmarks

The same queries, on DataFusion 54.0.0 through xarray_sql.Context and on DuckDB through xql.register, Intel i7-8700:

Query DataFusion operators DuckDB operators DataFusion DuckDB
1-layer GPT loss, batch 4 (112 lines of SQL) 12,166 120 8.4 s 0.055 s
GraphCast-shaped forecast, 1 processor round 1,816 243 43 s 17 s
GraphCast-shaped forecast, 2 processor rounds 5,948 282 159 s 13 s

A table read through xarray-sql is read again for every inlined copy. With ERA5 registered lazily from GCS, the 1-round forecast spent most of its time waiting on the network, so the benchmark loads its three time steps into memory first.

A training step through ddx (forward and backward) takes 2.7 s on DataFusion, less than the 8.4 s forward query alone, because ddx stores each step of its program as a table.

Upstream

DataFusion is working on this. apache/datafusion#25580 added WITH ... AS MATERIALIZED, in the PostgreSQL dialect only (apache/datafusion#17737), and apache/datafusion#26237 proposes deciding automatically, as PostgreSQL and DuckDB do. Neither is in DataFusion 54.

Options for xarray-sql

  • Once a DataFusion release has AS MATERIALIZED, check that it works with xarray-sql's table providers and the datafusion dialect Context uses, and document it.
  • Follow apache/datafusion#26237, which would fix this without changing queries.
  • Until then, document the behavior in docs/limitations.md, with the workaround: register an intermediate result as a table.
主要語言
Python
星號
142
分支
19
平均合併
1 天 9 小時
30 天內合併 PR
9

環境準備

  • 沒有 Dockerfile 或 Docker Compose 檔案
  • 沒有 Pull Request 範本
  • 閱讀貢獻指南

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

xqlsystems/xarray-sql 的其他 Issue

查看 xqlsystems/xarray-sql 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。