Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Native blocks without a JVM input publish SQL metrics on every batch, ignoring spark.comet.metrics.updateInterval

Abierto Apto para principiantes
  • ¿Algo no está bien?
#6,313 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
78/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust
Área
backend, performance

Línea de trabajo

Comience en native/core/src/execution/jni_api.rs, en la rama batch_receiver, y compare su ruta de publicación de métricas con la ruta de entrada de JVM. Reprodúzcalo con spark.comet.metrics.updateInterval=-1 y spark.comet.batchSize=1000; después, verifique que los bloques puramente nativos publiquen métricas solo en el intervalo configurado y que releasePlan publique los valores finales.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug performance priority:medium regression
Describe the bug

When a native block has no JVM input, meaning every leaf is a native scan, executePlan publishes the whole metric tree to the JVM after every output batch (the batch_receiver branch in native/core/src/execution/jni_api.rs). Blocks with a JVM input publish only after spark.comet.metrics.updateInterval has passed. So pure-native blocks ignore the setting, including a negative value, which the config doc says means "metrics will be updated upon task completion".

This came in with #3553 (0.14.0), which moved pure-native blocks onto a channel. Before that, both kinds of block checked the interval.

Each publish walks the plan, runs aggregate_by_name over every node's metrics, encodes a protobuf, and calls into the JVM, which decodes it and sets each SQLMetric. A native Parquet scan registers about 45 metrics for each file it opens, so the cost grows with the number of files a task has read.

I timed update_metrics in place on a release build (Apple M3 Max, local[1], 16M rows in 2,048 output batches from one task):

Plan Cost per publish Per task
Scan only, 1 file 19 µs 40 ms
Scan + filter + project 30 µs 62 ms
Scan only, 64 files in one task (2,905 raw metrics) 38 µs 78 ms
Scan + filter + project, noop write 26 µs 54 ms

Switching that branch to the interval check gave these medians of 7 runs:

Plan Wall time Process CPU time
Scan only, 1 file 174 → 143 ms 343 → 209 ms
Scan + filter + project 714 → 716 ms 882 → 834 ms
Scan only, 64 files in one task 210 → 167 ms 313 → 220 ms
Scan + filter + project, noop write 917 → 848 ms 1608 → 1505 ms

The scan + filter + project wall time doesn't move because the producer is the bottleneck there, and the publish runs on the consumer thread. It still spends CPU that other tasks on the executor could use.

This affects pure-native blocks whose output reaches the JVM batch by batch: a scan feeding a Spark write or a collect, a broadcast build side, JVM shuffle, or a fallback operator. A block that ends in a native shuffle write hands back only one batch, so it isn't affected.

Steps to reproduce

Set spark.comet.metrics.updateInterval=-1 and spark.comet.batchSize=1000, and read a 10,000-row Parquet file in one task. Inside the task, read the native scan's output_rows metric after the first batch. It is already non-zero, when it should stay 0 until the iterator closes.

Expected behavior

Pure-native blocks publish on the configured interval, as blocks with a JVM input do, and releasePlan publishes the final values.

Additional context

Found while looking at #1381.

Lenguaje dominante
Scala
Estrellas
1.3k
Forks
377
Merge medio
2 d 10 h
PR fusionados (30 d)
282

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-comet

Todos los issues de apache/datafusion-comet

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.