perf: filter evaluation in filtered reads runs on one thread

Abierto
#10,874 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
rust
Área
data, performance

Línea de trabajo

Start in rust/lance/src/io/exec/filtered_read.rs and trace how wrap_with_filter calls batch_filter in the try_buffered stream; compare that with the spawned decode work and lance_core::utils::tokio::spawn_cpu. Preserve batch order, the soft limit, and early-stop behavior. Run the order, limit, and early-stop tests in io::exec::filtered_read, then the scanner suites; done means they pass and filter evaluation can use multiple cores.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

performance

Summary

CPU-heavy filters run on a single thread in the filtered read, whatever the number of fragments or available cores. Examples: JSONPath (json_exists), regex, LIKE on large strings, ST_*, and higher-order functions.

In rust/lance/src/io/exec/filtered_read.rs, batch decode runs on spawned tasks, but wrap_with_filter applies datafusion_physical_plan::filter::batch_filter(batch, &filter) inside a .map() on each batch's future. That closure runs on whichever task polls the try_buffered stream, so filter evaluation is serialized onto one thread.

Evidence

On an Apple M4 Pro (14 cores), 1M rows, a full-scan filter json_exists(detections, '$[*] ? (@.label == "car" && @.confidence > .9)') gave:

Fragments Wall (p50) Cores busy
1 4.34 s 1.02
12 4.30 s 1.01
16 4.35 s 0.98

At 10M rows the same count takes about 41 s, still on one core. A decode-heavy read of a list<struct> column on the same machine, with a cheap filter, keeps about 4 cores busy, because decode is spawned.

Proposal

Evaluate the filter inside the spawned decode work, for example via lance_core::utils::tokio::spawn_cpu, keeping batch order, the soft limit and early-stop behaviour. We prototyped this behind an environment switch (inline vs spawn_cpu). The order, limit and early-stop tests in io::exec::filtered_read, and the scanner suites, pass in both modes. We'll add A/B timings here. We can open a PR.

Lenguaje dominante
Rust
Estrellas
7.3k
Forks
929
Merge medio
2 d 20 h
PR fusionados (30 d)
575

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de lance-format/lance

Todos los issues de lance-format/lance

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.