Where should shared CTE execution, compact arrays and GPU-resident training live?

Đang mở
  • Có gì không ổn?
#307 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 3 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
20/100
Loại issue
Tính năng
Độ rõ ràng
Cần làm rõ
Mức độ hoạt động
Sôi nổi
Công nghệ
postgresql, python, pytorch, sql
Lĩnh vực
backend, data, machine-learning

Hướng nghiên cứu

Read the linked work in #302, #303, and #305, along with DataFusion issue #267 and einfold issues #156 and #157. The issue asks maintainers to decide where shared CTE execution, compact arrays, and GPU-resident training belong, and what public APIs they need; it does not define an implementation. A useful outcome is an agreed ownership and API plan that respects the stated constraints.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

🌀 Claude: The nanoGPT performance work in #302, #303 and #305 (with einfold#156 and einfold#157) shows that a training step written in SQL can run within 3.76× of PyTorch on Apple's GPU. That is a strong result. This issue is for deciding where each piece of that work should live, so it can land without blurring the layers the projects split between them:

  • xarray-sql adapts xarray Datasets to SQL engines: registering them as tables, and turning query results back into Datasets.
  • ddx (https://github.com/xqlsystems/ddx) differentiates SQL.
  • einfold (https://github.com/xqlsystems/einfold) is a query optimizer that finds array computations in a plan and runs array algorithms for them.
  • DataFusion (https://datafusion.apache.org/) plans and runs the SQL. Computing a CTE read twice only once is its job, and it's tracked in #267 and on DataFusion's roadmap.

The pieces, and a proposed home for each

Piece In Proposed home
Prepared gradients, Context(threads=), faster registration #301 xarray-sql: fits as it is
SharedExecutor: materializes each CTE once, with a SQL parser and a function whitelist #302 DataFusion (#267); until then, an opt-in outside xarray-sql's core
MLX kernels for einfold's dense backend #302 einfold, or a small package of its own, behind einfold#156's backend interface
Compact arrays: DenseProgram in einfold, DenseExecutor in xarray-sql #303, einfold#157 einfold, likely in Rust beside its existing kernels, with a public interface
TrainingSession: device-resident state, compiling the whole step, publishing updates atomically, checkpoints #305 A separate package on public APIs. It is a training framework, outside xarray-sql's scope
The nanoGPT Shakespeare recipe, checked against PyTorch step by step #305 xarray-sql: as demo 3, on public APIs

Constraints to keep

  • No private imports across repositories. Today, xarray-sql would import einfold._native, the private names of einfold.dense_program (_Tensor, _expr, _Decline, ...) and ddxdb._ddxdb.to_sql. Any refactor of those would break xarray-sql. Each piece should depend only on its neighbours' public APIs.
  • Released dependencies, not forks. accelerators/*/build.py clones and installs pinned commits of personal forks of ddx and einfold. Those should become releases once the upstream PRs land.
  • The demos stay plain, legible SQL. Engine-motivated rewrites (for example, demo 2's grouped_softmax) should be fixed in the engine rather than in the demo's SQL.

Open questions

  1. Should einfold own a second plan executor (DenseProgram), or should compact arrays extend its Rust kernels? (For the einfold maintainers.)
  2. Until DataFusion shares CTEs, is an opt-in SharedExecutor worth carrying, and where? Demo 4 measured 11–15× from materializing each view, so the cost of waiting is real.
  3. What public interface would TrainingSession need from einfold and ddx to live in its own package?
Ngôn ngữ chính
Python
Star
142
Fork
19
Merge trung bình
3 ngày 47 phút
Pull request đã merge (30 ngày)
8

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của xqlsystems/xarray-sql

Tất cả issue của xqlsystems/xarray-sql

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.