Where should shared CTE execution, compact arrays and GPU-resident training live?

Abierto
  • ¿Algo no está bien?
#307 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
20/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
postgresql, python, pytorch, sql
Área
backend, data, machine-learning

Línea de trabajo

Read the linked work in #302, #303, and #305, along with DataFusion issue #267 and einfold issues #156 and #157. The issue asks maintainers to decide where shared CTE execution, compact arrays, and GPU-resident training belong, and what public APIs they need; it does not define an implementation. A useful outcome is an agreed ownership and API plan that respects the stated constraints.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

🌀 Claude: The nanoGPT performance work in #302, #303 and #305 (with einfold#156 and einfold#157) shows that a training step written in SQL can run within 3.76× of PyTorch on Apple's GPU. That is a strong result. This issue is for deciding where each piece of that work should live, so it can land without blurring the layers the projects split between them:

  • xarray-sql adapts xarray Datasets to SQL engines: registering them as tables, and turning query results back into Datasets.
  • ddx (https://github.com/xqlsystems/ddx) differentiates SQL.
  • einfold (https://github.com/xqlsystems/einfold) is a query optimizer that finds array computations in a plan and runs array algorithms for them.
  • DataFusion (https://datafusion.apache.org/) plans and runs the SQL. Computing a CTE read twice only once is its job, and it's tracked in #267 and on DataFusion's roadmap.

The pieces, and a proposed home for each

Piece In Proposed home
Prepared gradients, Context(threads=), faster registration #301 xarray-sql: fits as it is
SharedExecutor: materializes each CTE once, with a SQL parser and a function whitelist #302 DataFusion (#267); until then, an opt-in outside xarray-sql's core
MLX kernels for einfold's dense backend #302 einfold, or a small package of its own, behind einfold#156's backend interface
Compact arrays: DenseProgram in einfold, DenseExecutor in xarray-sql #303, einfold#157 einfold, likely in Rust beside its existing kernels, with a public interface
TrainingSession: device-resident state, compiling the whole step, publishing updates atomically, checkpoints #305 A separate package on public APIs. It is a training framework, outside xarray-sql's scope
The nanoGPT Shakespeare recipe, checked against PyTorch step by step #305 xarray-sql: as demo 3, on public APIs

Constraints to keep

  • No private imports across repositories. Today, xarray-sql would import einfold._native, the private names of einfold.dense_program (_Tensor, _expr, _Decline, ...) and ddxdb._ddxdb.to_sql. Any refactor of those would break xarray-sql. Each piece should depend only on its neighbours' public APIs.
  • Released dependencies, not forks. accelerators/*/build.py clones and installs pinned commits of personal forks of ddx and einfold. Those should become releases once the upstream PRs land.
  • The demos stay plain, legible SQL. Engine-motivated rewrites (for example, demo 2's grouped_softmax) should be fixed in the engine rather than in the demo's SQL.

Open questions

  1. Should einfold own a second plan executor (DenseProgram), or should compact arrays extend its Rust kernels? (For the einfold maintainers.)
  2. Until DataFusion shares CTEs, is an opt-in SharedExecutor worth carrying, and where? Demo 4 measured 11–15× from materializing each view, so the cost of waiting is real.
  3. What public interface would TrainingSession need from einfold and ddx to live in its own package?
Lenguaje dominante
Python
Estrellas
142
Forks
19
Merge medio
3 d 47 min
PR fusionados (30 d)
8

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de xqlsystems/xarray-sql

Todos los issues de xqlsystems/xarray-sql

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.