feat(amber): columnar wire format, operator contract, scan and filter
@Ma77Ball ya está trabajando en esto.
Desde el 17/9/2026.
Evaluación
Este issue todavía no se ha evaluado.
Descripción
Feature Summary
In one sentence: lay the foundation of columnar execution, the Arrow wire format and the opt-in operator contract, and prove it end to end with two operators (the CSV scan and the filter).
Parent: #8556 (opt-in columnar execution). This is PR 1 of the stacked series and pairs with apache/texera#8558.
What is this?
Before any operator can be made faster, two things must exist: a way to carry a batch of columns between workers, and a contract that lets an operator opt in to reading those columns. This issue adds both, then wires up the two simplest operators (scan and filter) so the whole path can be tested honestly.
Think of it as laying one lane of a new highway and driving two cars down it, while the old road stays open and default.
Proposed Solution or Design
The two new primitives.
ColumnarFrame: a data payload that carries one Arrow batch (the raw column bytes) plus a row count. It rides alongside the existing per-rowDataFrame.ColumnarOperatorExecutor/ColumnarResult: the opt-in contract. An operator can implementprocessColumnarBatch(...)and answerEmit(I made a new batch),Consumed(I absorbed it), orUnsupported(decode it and run the row path).
%%{init: {'theme':'dark', 'themeVariables': {'background':'#000000','lineColor':'#0F766E'}}}%%
flowchart LR
SCAN[CSV scan: read file straight into Arrow columns] --> WIRE[ColumnarFrame on the wire]
WIRE --> FILT[filter: keep rows on the column, no per-row decode]
FILT --> TERM[terminal]
classDef default fill:#000,color:#fff,stroke:#888,stroke-width:1px
How the engine routes a batch. The DataProcessor (the per-worker loop that runs the operator) checks whether the operator understands columns; if not, it decodes to rows. OutputManager, the partitioner, and the input side carry the batch end to end.
%%{init: {'theme':'dark', 'themeVariables': {'background':'#000000','lineColor':'#000000'}}}%%
flowchart TD
F{ColumnarFrame and<br/>operator opts in?} -->|yes| C[processColumnarBatch]
F -->|no| R[decode to rows, row path]
classDef default fill:#000,color:#fff,stroke:#888,stroke-width:1px
style C stroke:#1B7F3B
style R stroke:#B0451E
What lands here. ColumnarFrame, the contract, ArrowUtils (serialize/deserialize a batch), engine plumbing (DataProcessor, OutputManager, partitioner, input side), the Arrow-producing CSVScanSourceOpExec, and the Arrow-consuming SpecializedFilterOpExec. All behind a flag; default off.
| Row path (default) | Columnar path (flag on) | |
|---|---|---|
| scan output | one Tuple per row |
Arrow batch of columns |
| filter | evaluate per row | keep-mask over a column |
| between workers | N envelopes | 1 Arrow buffer |
Verified: the Arrow round-trip is lossless across all types, the vectorized filter matches the row filter exactly, and the existing DataProcessingSpec passes with the flag on and off.
High-level overview. Part of #8556.
- Lenguaje dominante
- Scala
- Estrellas
- 316
- Forks
- 189
- Merge medio
- 3 d 53 min
- PR fusionados (30 d)
- 181
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de apache/texera
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 90/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 90/100
Todos los issues de apache/texera
Issues similares
-
Area: Excel support
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
orbeon/orbeon-forms#7893 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
chipsalliance/rocket-chip#3833 ·
-
ShuffleManagerRegistry.register recursion guard is inverted, permits GlutenShuffleManager subclasses Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 65/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100