Where should shared CTE execution, compact arrays and GPU-resident training live?

未关闭
  • 哪里不对？
#307 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 3 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
20/100
Issue 类型
功能
描述清晰度
需要澄清
活跃度
活跃
技术栈
postgresql, python, pytorch, sql
领域
backend, data, machine-learning

调研方向

Read the linked work in #302, #303, and #305, along with DataFusion issue #267 and einfold issues #156 and #157. The issue asks maintainers to decide where shared CTE execution, compact arrays, and GPU-resident training belong, and what public APIs they need; it does not define an implementation. A useful outcome is an agreed ownership and API plan that respects the stated constraints.

由索引模型根据 Issue 内容生成。

描述

🌀 Claude: The nanoGPT performance work in #302, #303 and #305 (with einfold#156 and einfold#157) shows that a training step written in SQL can run within 3.76× of PyTorch on Apple's GPU. That is a strong result. This issue is for deciding where each piece of that work should live, so it can land without blurring the layers the projects split between them:

  • xarray-sql adapts xarray Datasets to SQL engines: registering them as tables, and turning query results back into Datasets.
  • ddx (https://github.com/xqlsystems/ddx) differentiates SQL.
  • einfold (https://github.com/xqlsystems/einfold) is a query optimizer that finds array computations in a plan and runs array algorithms for them.
  • DataFusion (https://datafusion.apache.org/) plans and runs the SQL. Computing a CTE read twice only once is its job, and it's tracked in #267 and on DataFusion's roadmap.

The pieces, and a proposed home for each

Piece In Proposed home
Prepared gradients, Context(threads=), faster registration #301 xarray-sql: fits as it is
SharedExecutor: materializes each CTE once, with a SQL parser and a function whitelist #302 DataFusion (#267); until then, an opt-in outside xarray-sql's core
MLX kernels for einfold's dense backend #302 einfold, or a small package of its own, behind einfold#156's backend interface
Compact arrays: DenseProgram in einfold, DenseExecutor in xarray-sql #303, einfold#157 einfold, likely in Rust beside its existing kernels, with a public interface
TrainingSession: device-resident state, compiling the whole step, publishing updates atomically, checkpoints #305 A separate package on public APIs. It is a training framework, outside xarray-sql's scope
The nanoGPT Shakespeare recipe, checked against PyTorch step by step #305 xarray-sql: as demo 3, on public APIs

Constraints to keep

  • No private imports across repositories. Today, xarray-sql would import einfold._native, the private names of einfold.dense_program (_Tensor, _expr, _Decline, ...) and ddxdb._ddxdb.to_sql. Any refactor of those would break xarray-sql. Each piece should depend only on its neighbours' public APIs.
  • Released dependencies, not forks. accelerators/*/build.py clones and installs pinned commits of personal forks of ddx and einfold. Those should become releases once the upstream PRs land.
  • The demos stay plain, legible SQL. Engine-motivated rewrites (for example, demo 2's grouped_softmax) should be fixed in the engine rather than in the demo's SQL.

Open questions

  1. Should einfold own a second plan executor (DenseProgram), or should compact arrays extend its Rust kernels? (For the einfold maintainers.)
  2. Until DataFusion shares CTEs, is an opt-in SharedExecutor worth carrying, and where? Demo 4 measured 11–15× from materializing each view, so the cost of waiting is real.
  3. What public interface would TrainingSession need from einfold and ddx to live in its own package?
主要语言
Python
星标
142
派生
19
平均合并
3 天 47 分钟
30 天内合并 PR
8

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

xqlsystems/xarray-sql 的其他 Issue

查看 xqlsystems/xarray-sql 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。