Where should shared CTE execution, compact arrays and GPU-resident training live?

Ouverte
  • Quelque chose ne va pas ?
#307 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 3 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
20/100
Type d'issue
Fonctionnalité
Clarté
À clarifier
Activité
Active
Stack technique
postgresql, python, pytorch, sql
Domaine
backend, data, machine-learning

Piste de recherche

Read the linked work in #302, #303, and #305, along with DataFusion issue #267 and einfold issues #156 and #157. The issue asks maintainers to decide where shared CTE execution, compact arrays, and GPU-resident training belong, and what public APIs they need; it does not define an implementation. A useful outcome is an agreed ownership and API plan that respects the stated constraints.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

🌀 Claude: The nanoGPT performance work in #302, #303 and #305 (with einfold#156 and einfold#157) shows that a training step written in SQL can run within 3.76× of PyTorch on Apple's GPU. That is a strong result. This issue is for deciding where each piece of that work should live, so it can land without blurring the layers the projects split between them:

  • xarray-sql adapts xarray Datasets to SQL engines: registering them as tables, and turning query results back into Datasets.
  • ddx (https://github.com/xqlsystems/ddx) differentiates SQL.
  • einfold (https://github.com/xqlsystems/einfold) is a query optimizer that finds array computations in a plan and runs array algorithms for them.
  • DataFusion (https://datafusion.apache.org/) plans and runs the SQL. Computing a CTE read twice only once is its job, and it's tracked in #267 and on DataFusion's roadmap.

The pieces, and a proposed home for each

Piece In Proposed home
Prepared gradients, Context(threads=), faster registration #301 xarray-sql: fits as it is
SharedExecutor: materializes each CTE once, with a SQL parser and a function whitelist #302 DataFusion (#267); until then, an opt-in outside xarray-sql's core
MLX kernels for einfold's dense backend #302 einfold, or a small package of its own, behind einfold#156's backend interface
Compact arrays: DenseProgram in einfold, DenseExecutor in xarray-sql #303, einfold#157 einfold, likely in Rust beside its existing kernels, with a public interface
TrainingSession: device-resident state, compiling the whole step, publishing updates atomically, checkpoints #305 A separate package on public APIs. It is a training framework, outside xarray-sql's scope
The nanoGPT Shakespeare recipe, checked against PyTorch step by step #305 xarray-sql: as demo 3, on public APIs

Constraints to keep

  • No private imports across repositories. Today, xarray-sql would import einfold._native, the private names of einfold.dense_program (_Tensor, _expr, _Decline, ...) and ddxdb._ddxdb.to_sql. Any refactor of those would break xarray-sql. Each piece should depend only on its neighbours' public APIs.
  • Released dependencies, not forks. accelerators/*/build.py clones and installs pinned commits of personal forks of ddx and einfold. Those should become releases once the upstream PRs land.
  • The demos stay plain, legible SQL. Engine-motivated rewrites (for example, demo 2's grouped_softmax) should be fixed in the engine rather than in the demo's SQL.

Open questions

  1. Should einfold own a second plan executor (DenseProgram), or should compact arrays extend its Rust kernels? (For the einfold maintainers.)
  2. Until DataFusion shares CTEs, is an opt-in SharedExecutor worth carrying, and where? Demo 4 measured 11–15× from materializing each view, so the cost of waiting is real.
  3. What public interface would TrainingSession need from einfold and ddx to live in its own package?
Langage dominant
Python
Étoiles
142
Forks
19
Merge moyen
3 j 47 min
PR mergées (30 j)
8

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de xqlsystems/xarray-sql

Toutes les issues de xqlsystems/xarray-sql

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.