Comet Native Executors: sample a range exchange without re-running the stages below it

Chiusa
  • Qualcosa non va?
#7,367 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust, scala
Ambito
backend, distributed-systems

Direzione di ricerca

Start with range_exchange in native/native-executors/src/assembly.rs and collect in src/frontend.rs to trace how sampling runs the stages below an exchange. Compare the proposed approaches, including materialized shuffle reuse and runtime sampling; done means range exchanges can be sampled without repeatedly rerunning lower stages. The issue does not select an approach, so clarify the design before implementation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement requires-triage
What is the problem the feature request solves?

To range-partition an exchange, the Comet scheduler (#7289) runs a sampling job of its own: a SampleExec over the stage that feeds the exchange. That is a separate Ballista job, so it runs that stage and every stage below it, then the query runs them all again. Every further range exchange above them adds another run. Spark's RangePartitioner also re-runs the child, but it reuses the shuffle outputs below it.

The sampling jobs also run one after another while the query is being planned, even for range exchanges whose inputs don't depend on each other.

Describe the potential solution

Options, roughly from least to most work:

  • Sample range exchanges whose inputs are independent concurrently.
  • Sample from the materialized shuffle output below the exchange, so only the producing stage runs twice.
  • Find the split points at run time, as Ballista's adaptive planner does for window functions: RuntimeStatsExec keeps a quantile sketch and UnorderedRangeRepartitionExec cuts by it. That covers only the first sort key today, so it would need compound keys. The driver-side sampler, SampleExec and its codec would then go away.
Additional context

The sampling is range_exchange in native/native-executors/src/assembly.rs and collect in src/frontend.rs.

Part of #7288.

Willingness to contribute

Yes. I can contribute this feature independently.

Lingua principale
Scala
Stelle
1.3k
Fork
405
Merge medio
3g 1h
PR unite (30g)
277

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-comet

Tutte le issue di apache/datafusion-comet

Issue simili

Altre issue su Scala

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.