Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Native blocks without a JVM input publish SQL metrics on every batch, ignoring spark.comet.metrics.updateInterval

Chiusa Adatta ai principianti
  • Qualcosa non va?
#6,313 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
78/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
backend, performance

Direzione di ricerca

Iniziate in native/core/src/execution/jni_api.rs, nel ramo batch_receiver, e confrontate il relativo percorso di pubblicazione delle metriche con il percorso di input JVM. Riproducete il problema con spark.comet.metrics.updateInterval=-1 e spark.comet.batchSize=1000, quindi verificate che i blocchi esclusivamente nativi pubblichino le metriche solo all’intervallo configurato e che releasePlan pubblichi i valori finali.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug performance priority:medium regression
Describe the bug

When a native block has no JVM input, meaning every leaf is a native scan, executePlan publishes the whole metric tree to the JVM after every output batch (the batch_receiver branch in native/core/src/execution/jni_api.rs). Blocks with a JVM input publish only after spark.comet.metrics.updateInterval has passed. So pure-native blocks ignore the setting, including a negative value, which the config doc says means "metrics will be updated upon task completion".

This came in with #3553 (0.14.0), which moved pure-native blocks onto a channel. Before that, both kinds of block checked the interval.

Each publish walks the plan, runs aggregate_by_name over every node's metrics, encodes a protobuf, and calls into the JVM, which decodes it and sets each SQLMetric. A native Parquet scan registers about 45 metrics for each file it opens, so the cost grows with the number of files a task has read.

I timed update_metrics in place on a release build (Apple M3 Max, local[1], 16M rows in 2,048 output batches from one task):

Plan Cost per publish Per task
Scan only, 1 file 19 µs 40 ms
Scan + filter + project 30 µs 62 ms
Scan only, 64 files in one task (2,905 raw metrics) 38 µs 78 ms
Scan + filter + project, noop write 26 µs 54 ms

Switching that branch to the interval check gave these medians of 7 runs:

Plan Wall time Process CPU time
Scan only, 1 file 174 → 143 ms 343 → 209 ms
Scan + filter + project 714 → 716 ms 882 → 834 ms
Scan only, 64 files in one task 210 → 167 ms 313 → 220 ms
Scan + filter + project, noop write 917 → 848 ms 1608 → 1505 ms

The scan + filter + project wall time doesn't move because the producer is the bottleneck there, and the publish runs on the consumer thread. It still spends CPU that other tasks on the executor could use.

This affects pure-native blocks whose output reaches the JVM batch by batch: a scan feeding a Spark write or a collect, a broadcast build side, JVM shuffle, or a fallback operator. A block that ends in a native shuffle write hands back only one batch, so it isn't affected.

Steps to reproduce

Set spark.comet.metrics.updateInterval=-1 and spark.comet.batchSize=1000, and read a 10,000-row Parquet file in one task. Inside the task, read the native scan's output_rows metric after the first batch. It is already non-zero, when it should stay 0 until the iterator closes.

Expected behavior

Pure-native blocks publish on the configured interval, as blocks with a JVM input do, and releasePlan publishes the final values.

Additional context

Found while looking at #1381.

Lingua principale
Scala
Stelle
1.3k
Fork
387
Merge medio
2g 6h
PR unite (30g)
419

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-comet

Tutte le issue di apache/datafusion-comet

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.