Where should shared CTE execution, compact arrays and GPU-resident training live?

Offen
  • Stimmt etwas nicht?
#307 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 3 Tagen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
20/100
Issue-Typ
Feature
Klarheit
Muss geklärt werden
Aktivitätsstatus
Aktiv
Tech-Stack
postgresql, python, pytorch, sql
Bereich
backend, data, machine-learning

Rechercherichtung

Read the linked work in #302, #303, and #305, along with DataFusion issue #267 and einfold issues #156 and #157. The issue asks maintainers to decide where shared CTE execution, compact arrays, and GPU-resident training belong, and what public APIs they need; it does not define an implementation. A useful outcome is an agreed ownership and API plan that respects the stated constraints.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

🌀 Claude: The nanoGPT performance work in #302, #303 and #305 (with einfold#156 and einfold#157) shows that a training step written in SQL can run within 3.76× of PyTorch on Apple's GPU. That is a strong result. This issue is for deciding where each piece of that work should live, so it can land without blurring the layers the projects split between them:

  • xarray-sql adapts xarray Datasets to SQL engines: registering them as tables, and turning query results back into Datasets.
  • ddx (https://github.com/xqlsystems/ddx) differentiates SQL.
  • einfold (https://github.com/xqlsystems/einfold) is a query optimizer that finds array computations in a plan and runs array algorithms for them.
  • DataFusion (https://datafusion.apache.org/) plans and runs the SQL. Computing a CTE read twice only once is its job, and it's tracked in #267 and on DataFusion's roadmap.

The pieces, and a proposed home for each

Piece In Proposed home
Prepared gradients, Context(threads=), faster registration #301 xarray-sql: fits as it is
SharedExecutor: materializes each CTE once, with a SQL parser and a function whitelist #302 DataFusion (#267); until then, an opt-in outside xarray-sql's core
MLX kernels for einfold's dense backend #302 einfold, or a small package of its own, behind einfold#156's backend interface
Compact arrays: DenseProgram in einfold, DenseExecutor in xarray-sql #303, einfold#157 einfold, likely in Rust beside its existing kernels, with a public interface
TrainingSession: device-resident state, compiling the whole step, publishing updates atomically, checkpoints #305 A separate package on public APIs. It is a training framework, outside xarray-sql's scope
The nanoGPT Shakespeare recipe, checked against PyTorch step by step #305 xarray-sql: as demo 3, on public APIs

Constraints to keep

  • No private imports across repositories. Today, xarray-sql would import einfold._native, the private names of einfold.dense_program (_Tensor, _expr, _Decline, ...) and ddxdb._ddxdb.to_sql. Any refactor of those would break xarray-sql. Each piece should depend only on its neighbours' public APIs.
  • Released dependencies, not forks. accelerators/*/build.py clones and installs pinned commits of personal forks of ddx and einfold. Those should become releases once the upstream PRs land.
  • The demos stay plain, legible SQL. Engine-motivated rewrites (for example, demo 2's grouped_softmax) should be fixed in the engine rather than in the demo's SQL.

Open questions

  1. Should einfold own a second plan executor (DenseProgram), or should compact arrays extend its Rust kernels? (For the einfold maintainers.)
  2. Until DataFusion shares CTEs, is an opt-in SharedExecutor worth carrying, and where? Demo 4 measured 11–15× from materializing each view, so the cost of waiting is real.
  3. What public interface would TrainingSession need from einfold and ddx to live in its own package?
Vorherrschende Sprache
Python
Sterne
142
Forks
19
Ø Merge
3 T. 47 Min.
Gemergte PRs (30 T.)
8

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus xqlsystems/xarray-sql

Alle Issues in xqlsystems/xarray-sql

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.