Context.prepare can't prepare a query that calls grad, vjp or jvp

Aberta
  • Algo errado?
#294 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 3 dias

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
25/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
python, sql
Domínio
backend, machine-learning

Direção de pesquisa

Start from Context.prepare() and Context.sql() in the xarray_sql package, and reproduce the ValueError from the snippet to see where DataFusion planning rejects grad. Then read xarray_sql.ad.sql_all and ddxdb's own prepare, and check how PreparedQuery handles writes into registered Datasets. Done means prepare() runs a query that calls grad, vjp or jvp and gives the same results as sql(). The interim NotImplementedError is tied to the unmerged #269 branch, so coordinate there first.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

ml

🌀 Claude: Context.sql() computes a query's grad, vjp and jvp calls through ddx (the ml extra, on the ml-demos branch of #269), but Context.prepare() plans the query with DataFusion alone, so it fails on the call:

import numpy as np, xarray as xr, xarray_sql as xql

ctx = xql.Context()
ctx.from_dataset("w", xr.Dataset({"val": ("feature", np.array([1.0, 2.0]))}, coords={"feature": [0, 1]}))
q = ("WITH loss AS (SELECT SUM(val * val) AS l FROM w) "
     "SELECT feature, grad(loss, val) AS g FROM w ORDER BY feature")
ctx.sql(q).to_pandas()   # g = [2.0, 4.0]
ctx.prepare(q)           # ValueError: Error during planning: Invalid function 'grad'. Did you mean 'rpad'?

(ddxdb 0.4.6, DataFusion 55.)

A training loop is where prepare matters most: it runs the same gradient step on every iteration, and planning ddx's programs costs far more than running them at small sizes. ddxdb has its own prepare, which plans the forward and backward programs once.

Proposal:

  • Until this is done, #269 makes prepare() raise a NotImplementedError that names sql() and xarray_sql.ad.sql_all, rather than DataFusion's planning error.
  • Then route a query that calls grad, vjp or jvp through ddxdb's prepare, keeping PreparedQuery's contract: writes into registered Datasets before a call, and the check that a table hasn't been registered again since the query was prepared.
  • sql_all's several statements over one backward pass would want the same, as a prepare_all or a prepare that takes several statements.
Linguagem predominante
Python
Estrelas
142
Forks
19
Merge médio
3d 47min
PRs com merge (30d)
8

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de xqlsystems/xarray-sql

Todas as issues de xqlsystems/xarray-sql

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.