Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Perf: expose natural Xarray scan ordering to eliminate redundant SQL sorts

Aberta
  • Algo errado?
#249 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
28/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Pouca atividade
Stack de tecnologia
numpy, python, sql
Domínio
databases, performance

Direção de pesquisa

Start with the XarrayScanExec node and its PlanProperties in the native DataFusion provider, then reproduce the ordered EXPLAIN plan that shows SortExec above it. Check whether the scan can declare C-order within each partition and time-range ordering across partitions, so DataFusion can choose SortPreservingMergeExec. Done means ordered results stay exact and a regression test shows no SortExec for an order that matches the dataset's dimension order.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

🤖 Samudra's canonical data-loader experiment found that ORDER BY time, lat, lon is a material part of point-read latency even though XarrayScanExec already emits each dense block in C-order.

Local benchmark setup: an uncompressed float32 Zarr store with 32 times, 16 variables, a 180×360 grid, chunks=None at Xarray open, and XQL scan partitions of {"time": 1}. Selecting four times and all 16 variables took approximately:

  • ordered DataFusion collection: 21 ms
  • the same query without ORDER BY: 13 ms

The ordered physical plan contains:

SortPreservingMergeExec: time, lat, lon
  SortExec: time, lat, lon
    FilterExec: time IN (...)
      RepartitionExec
        XarrayScanExec

For an ML loader, relying on unspecified result order is not acceptable, and scattering unordered rows in NumPy erased the saved time. Could the native provider declare the scan's natural intra-partition ordering in PlanProperties, plus enough partition-range ordering for DataFusion to use a preserving merge instead of sorting the cells?

A useful regression test would assert both exact ordered results and the absence of SortExec in EXPLAIN for an order compatible with the dataset's dimension order. This may also reduce sort-buffer memory for dense-grid round trips.

Linguagem predominante
Python
Estrelas
142
Forks
19
Merge médio
1d 9h
PRs com merge (30d)
9

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de xqlsystems/xarray-sql

Todas as issues de xqlsystems/xarray-sql

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.