Comet Native Executors: pass scalar subquery results to the cluster without rewriting the plan

Abierto
  • ¿Algo no está bien?
#7,374 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 2 días

@andygrove ya está trabajando en esto.

Desde el 1/11/2026.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
rust, scala, spark
Área
data-engineering, distributed-systems

Línea de trabajo

Start by reading withSubqueryResults in CometNativeExecutorsExec.scala, then trace how QueryRequest and Fragment carry data to PhysicalPlanner. Check how ExprStruct::Subquery is handled when planning blocks with native inputs. Done means subquery results are sent once and the planner can look them up without rewriting serialized plans.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement
What is the problem the feature request solves?

Before submitting a query, the driver (#7289) replaces each scalar subquery in the plan with a literal of its result. It parses the serialized plan of every stage that holds a subquery, and every partition's own plan for a stage that reads files, walks them, and serializes them again. For a scan over many files, that is the bulk of the request, redone on the driver for every query.

Describe the potential solution

Send the results once, as map<int64, Expr> subquery_results on QueryRequest and Fragment. Have PhysicalPlanner look up ExprStruct::Subquery in that map when it plans a block with native inputs, where today it fails because there is no JVM. The driver would then serialize only the literals.

Additional context

The rewrite is withSubqueryResults in CometNativeExecutorsExec.scala.

Part of #7288.

Willingness to contribute

Yes. I can contribute this feature independently.

Lenguaje dominante
Scala
Estrellas
1.3k
Forks
405
Merge medio
3 d 4 min
PR fusionados (30 d)
282

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-comet

Todos los issues de apache/datafusion-comet

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.