Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

feat(amber): columnar hardening (versioning, compression, sink)

Open
  • Is something wrong?
#8,567 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@Ma77Ball is already working on this.

Since Sep 17, 2026.

Assessment

This issue has not been assessed yet.

Description

Feature Summary

In one sentence: harden the columnar path for real use: version the wire format, add optional compression, offer a vectorized write-to-Iceberg sink, and speed up the row decode for the cases that still need rows.

Parent: #8556 (opt-in columnar execution). PR 4 (final) of the stacked series; pairs with apache/texera#8561.

What is this?

The earlier PRs made columnar execution work. This one makes it safe to evolve and cheaper at the edges: a version stamp so old and new workers never misread each other, optional compression to shrink what crosses the network, a sink that writes Arrow straight to storage, and faster decode helpers for the fallback path.

Proposed Solution or Design

Wire versioning. ColumnarFrame now carries a format version. A worker that sees a version it does not support rejects it loudly instead of quietly misreading the bytes. This is the seatbelt that lets the format change later without silent corruption.

%%{init: {'theme':'dark', 'themeVariables': {'background':'#000000','lineColor':'#000000'}}}%%
flowchart TD
  IN[ColumnarFrame v=N] --> Q{N supported?}
  Q -->|yes| GO[read batch]
  Q -->|no| ERR[reject with a clear error]
  classDef default fill:#000,color:#fff,stroke:#888,stroke-width:1px
  style GO stroke:#1B7F3B
  style ERR stroke:#B0451E

Optional wire compression. LZ4_FRAME or ZSTD via arrow-compression, config-gated. The reader auto-detects compressed versus uncompressed, so a compressed sender and a plain sender interoperate.

Vectorized Iceberg sink. IcebergTableWriter.writeArrowBatch writes an Arrow batch straight to Iceberg through a lazy ArrowRecordView that reads values from the Arrow columns, materializing no intermediate rows. Gated by enable-vectorized-sink.

%%{init: {'theme':'dark', 'themeVariables': {'background':'#000000','lineColor':'#0F766E'}}}%%
flowchart LR
  A[Arrow batch] --> V[ArrowRecordView: read straight from columns]
  V --> ICE[Iceberg data file]
  classDef default fill:#000,color:#fff,stroke:#888,stroke-width:1px

Faster decode. Schema-once and projected decode helpers in ArrowUtils, so the fallback-to-rows path pays less.

Concern Before After
format change silent misread risk version stamp, explicit reject
network size uncompressed optional LZ4/ZSTD
sink write decode to rows first Arrow straight to Iceberg
decode resolve schema per row resolve once, decode only needed columns

Verified: batches round-trip unchanged including compressed and versioned frames; 2-worker shuffle row == columnar with the vectorized sink on.

High-level overview. Part of #8556.

Dominant language
Scala
Stars
316
Forks
192
Avg merge
4d 17h
Merged PRs (30d)
141

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/texera

All issues in apache/texera

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.