Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Perf: coordinate-only projections skip pushdown, so every data variable is still loaded

Aperta Adatta ai principianti
  • Qualcosa non va?
#237 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
72/100
Tipo di issue
Refactoring
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
numpy, pandas, python, rust
Ambito
data-engineering

Direzione di ricerca

Il predicato che decide il pushdown si trova in src/lib.rs in PrunableStreamingTable::scan() (intorno alle righe 607-614). Leggi il ramo corrispondente della factory in read_xarray_table e il ramo senza variabili dati di iter_record_batches, poi controlla i contatori delle chiamate del reproducer. È completato quando SELECT lat, lon passa alla factory una proiezione non None, non legge chunk di variabili dati e restituisce batch corrispondenti alla scansione senza proiezione, coperto da un nuovo test.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

PrunableStreamingTable::scan() only pushes the projection down to the Python partition factory when the projection contains at least one data variable. A projection made purely of dimension coordinates falls through to the non-pushdown branch, the factory is called with None, and every data variable in every scanned partition is loaded from storage before ProjectionExec throws it away.

src/lib.rs (main @ 5d7633e, lines 607 to 614):

//   - projection contains only dimension columns (ds.to_dataframe()
//     needs at least one data variable; dimensions are always loaded)
let push_projection = match projection {
    Some(indices) if !indices.is_empty() => indices
        .iter()
        .any(|&i| !self.dimension_columns.contains(self.schema.field(i).name())),
    _ => false,
};

The reason given in that comment no longer holds. The factory built by read_xarray_table already handles a coordinate-only projection explicitly:

if data_vars_needed:
    ds_block = ds[data_vars_needed].isel(block)
else:
    # Only dimension coords requested - drop all data vars to avoid
    # loading them unnecessarily (e.g. for queries like SELECT lat, lon).
    ds_block = ds.drop_vars(list(ds.data_vars)).isel(block)

and iter_record_batches has a matching if ds.data_vars: ... else: ... branch that takes its shape from ds.sizes when the Dataset has no data variables. So the Python side is ready for this case; only the Rust predicate keeps it unreachable.

Reproducer

Counts real data-variable chunk reads with a dask.array.map_blocks counter, and records the projection the factory receives.

import dask, dask.array as da, numpy as np, pandas as pd, xarray as xr
from datafusion import SessionContext
from xarray_sql import read_xarray_table

dask.config.set(scheduler="synchronous")
loads = {"n": 0}

def block(block_info=None):
    loads["n"] += 1                       # one call == one data-var chunk read
    return np.zeros(block_info[None]["chunk-shape"], dtype="float32")

ds = xr.Dataset(
    {
        name: (("time", "lat", "lon"),
               da.map_blocks(block, dtype="float32",
                             chunks=((10,) * 4, (20,), (30,))))
        for name in ("air", "hum")
    },
    coords={"time": pd.date_range("2000-01-01", periods=40),
            "lat": np.linspace(-90, 90, 20),
            "lon": np.linspace(0, 359, 30)},
)

seen = []
ctx = SessionContext()
ctx.register_table("t", read_xarray_table(
    ds, {"time": 10}, _iteration_callback=lambda blk, proj: seen.append(proj)))

for sql in ("SELECT COUNT(*) c FROM t",
            "SELECT lat, air FROM t",
            "SELECT DISTINCT lat FROM t",
            "SELECT lat, lon FROM t"):
    loads["n"], seen[:] = 0, []
    ctx.sql(sql).to_pandas()
    proj = str(seen[0]) if seen else "(no scan)"
    print(f"{sql:28} projection={proj:22} chunks_loaded={loads['n']}")

Output on main (4 partitions, 2 data variables):

SELECT COUNT(*) c FROM t     projection=(no scan)              chunks_loaded=0
SELECT lat, air FROM t       projection=['lat', 'air']         chunks_loaded=4
SELECT DISTINCT lat FROM t   projection=None                   chunks_loaded=8
SELECT lat, lon FROM t       projection=None                   chunks_loaded=8

SELECT lat, air is correct: pushdown happens and only air is read, 4 chunks. The two coordinate-only queries read 8 chunks, both data variables across all four partitions, for an answer that needs zero data-variable bytes.

Impact

Any query answered entirely from the coordinate axes still pays for the full data payload:

  • SELECT DISTINCT time FROM ds or SELECT DISTINCT level FROM ds to discover what an axis contains
  • SELECT lat, lon FROM ds WHERE ... to fetch a grid outline
  • COUNT(DISTINCT lat)
  • any front end populating filter widgets from the dimension coordinates

On a local NetCDF this is a wasted copy. On a remote Zarr store it is a chunk fetch per data variable per partition. The waste also scales with the number of data variables in the table, which the same-dimensions grouping in from_dataset makes the common case.

Proposed fix

Push the projection whenever it is Some and non-empty, dropping the "contains at least one data variable" condition, and update the comment. The non-pushdown branch stays for None and Some([]).

I checked the Python half directly rather than assuming it: feeding iter_record_batches a block with drop_vars(list(ds.data_vars)) and a coords-only pa.schema returns the right row count and columns, so the projected path looks ready for this. I have not built the Rust change yet, so the predicate edit still needs compiling and testing.

Worth adding a test that asserts the factory receives the coordinate-only projection and that the resulting batches match the unprojected scan.

Parent: #126

Lingua principale
Python
Stelle
142
Fork
19
Merge medio
1g 9h
PR unite (30g)
10

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di xqlsystems/xarray-sql

Tutte le issue di xqlsystems/xarray-sql

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.