Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

feat(amber): columnar hardening (versioning, compression, sink)

Abierto
  • ¿Algo no está bien?
#8,567 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@Ma77Ball ya está trabajando en esto.

Desde el 17/9/2026.

  • #8561 de @Ma77Ball — abierto

Evaluación

Este issue todavía no se ha evaluado.

Descripción

Feature Summary

In one sentence: harden the columnar path for real use: version the wire format, add optional compression, offer a vectorized write-to-Iceberg sink, and speed up the row decode for the cases that still need rows.

Parent: #8556 (opt-in columnar execution). PR 4 (final) of the stacked series; pairs with apache/texera#8561.

What is this?

The earlier PRs made columnar execution work. This one makes it safe to evolve and cheaper at the edges: a version stamp so old and new workers never misread each other, optional compression to shrink what crosses the network, a sink that writes Arrow straight to storage, and faster decode helpers for the fallback path.

Proposed Solution or Design

Wire versioning. ColumnarFrame now carries a format version. A worker that sees a version it does not support rejects it loudly instead of quietly misreading the bytes. This is the seatbelt that lets the format change later without silent corruption.

%%{init: {'theme':'dark', 'themeVariables': {'background':'#000000','lineColor':'#000000'}}}%%
flowchart TD
  IN[ColumnarFrame v=N] --> Q{N supported?}
  Q -->|yes| GO[read batch]
  Q -->|no| ERR[reject with a clear error]
  classDef default fill:#000,color:#fff,stroke:#888,stroke-width:1px
  style GO stroke:#1B7F3B
  style ERR stroke:#B0451E

Optional wire compression. LZ4_FRAME or ZSTD via arrow-compression, config-gated. The reader auto-detects compressed versus uncompressed, so a compressed sender and a plain sender interoperate.

Vectorized Iceberg sink. IcebergTableWriter.writeArrowBatch writes an Arrow batch straight to Iceberg through a lazy ArrowRecordView that reads values from the Arrow columns, materializing no intermediate rows. Gated by enable-vectorized-sink.

%%{init: {'theme':'dark', 'themeVariables': {'background':'#000000','lineColor':'#0F766E'}}}%%
flowchart LR
  A[Arrow batch] --> V[ArrowRecordView: read straight from columns]
  V --> ICE[Iceberg data file]
  classDef default fill:#000,color:#fff,stroke:#888,stroke-width:1px

Faster decode. Schema-once and projected decode helpers in ArrowUtils, so the fallback-to-rows path pays less.

Concern Before After
format change silent misread risk version stamp, explicit reject
network size uncompressed optional LZ4/ZSTD
sink write decode to rows first Arrow straight to Iceberg
decode resolve schema per row resolve once, decode only needed columns

Verified: batches round-trip unchanged including compressed and versioned frames; 2-worker shuffle row == columnar with the vectorized sink on.

High-level overview. Part of #8556.

Lenguaje dominante
Scala
Estrellas
321
Forks
192
Merge medio
2 d 8 h
PR fusionados (30 d)
131

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/texera

Todos los issues de apache/texera

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.