feat(amber): columnar hardening (versioning, compression, sink)
维护者通常 1 天内回复
评估
这个 Issue 还没有评估数据。
描述
Feature Summary
In one sentence: harden the columnar path for real use: version the wire format, add optional compression, offer a vectorized write-to-Iceberg sink, and speed up the row decode for the cases that still need rows.
Parent: #8556 (opt-in columnar execution). PR 4 (final) of the stacked series; pairs with apache/texera#8561.
What is this?
The earlier PRs made columnar execution work. This one makes it safe to evolve and cheaper at the edges: a version stamp so old and new workers never misread each other, optional compression to shrink what crosses the network, a sink that writes Arrow straight to storage, and faster decode helpers for the fallback path.
Proposed Solution or Design
Wire versioning. ColumnarFrame now carries a format version. A worker that sees a version it does not support rejects it loudly instead of quietly misreading the bytes. This is the seatbelt that lets the format change later without silent corruption.
%%{init: {'theme':'dark', 'themeVariables': {'background':'#000000','lineColor':'#000000'}}}%%
flowchart TD
IN[ColumnarFrame v=N] --> Q{N supported?}
Q -->|yes| GO[read batch]
Q -->|no| ERR[reject with a clear error]
classDef default fill:#000,color:#fff,stroke:#888,stroke-width:1px
style GO stroke:#1B7F3B
style ERR stroke:#B0451E
Optional wire compression. LZ4_FRAME or ZSTD via arrow-compression, config-gated. The reader auto-detects compressed versus uncompressed, so a compressed sender and a plain sender interoperate.
Vectorized Iceberg sink. IcebergTableWriter.writeArrowBatch writes an Arrow batch straight to Iceberg through a lazy ArrowRecordView that reads values from the Arrow columns, materializing no intermediate rows. Gated by enable-vectorized-sink.
%%{init: {'theme':'dark', 'themeVariables': {'background':'#000000','lineColor':'#0F766E'}}}%%
flowchart LR
A[Arrow batch] --> V[ArrowRecordView: read straight from columns]
V --> ICE[Iceberg data file]
classDef default fill:#000,color:#fff,stroke:#888,stroke-width:1px
Faster decode. Schema-once and projected decode helpers in ArrowUtils, so the fallback-to-rows path pays less.
| Concern | Before | After |
|---|---|---|
| format change | silent misread risk | version stamp, explicit reject |
| network size | uncompressed | optional LZ4/ZSTD |
| sink write | decode to rows first | Arrow straight to Iceberg |
| decode | resolve schema per row | resolve once, decode only needed columns |
Verified: batches round-trip unchanged including compressed and versioned frames; 2-worker shuffle row == columnar with the vectorized sink on.
High-level overview. Part of #8556.
- 主要语言
- Scala
- 星标
- 322
- 派生
- 195
- 平均合并
- 2 天 16 小时
- 30 天内合并 PR
- 130
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
apache/texera 的其他 Issue
-
Search bar autocomplete stops working after a model appears in the suggestions可能已有人在做 @aglinxinyuan 于 23 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
维护者通常 1 天内回复
-
JSONL File Scan reads a JSON null as the text "null"可能已有人在做 @CaroFernando 于 21 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 86/100
apache/texera#8674 · 1 条评论 · 已指派 1 人 ·
维护者通常 1 天内回复
-
LARGE_BINARY test leaves a real S3 object behind on every S3-reachable run可能已有人在做 @ShawnGarello 于 6 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 84/100
维护者通常 1 天内回复
-
Missing workflow returns 500 from workflow type endpoint可能重新可做 @VuMartin 于 46 天前认领，目前没有进行中的 PR。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 75/100
propensive/soundness#2294 ·
维护者通常 1 天内回复
-
area:scale audit priority:P2
难度 2/5 1-3 小时 新手友好度 76/100
KD5UJC-XYZ/scarab#231 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 70/100
SpinalHDL/VexiiRiscv#198 ·
维护者通常 3 天内回复
-
nativeLink relinks every build: IO.write leaves stale trailing bytes in build-checksum可能已有人在做 @kimmoeklund 于 1 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
scala-native/scala-native#5111 ·
维护者通常 1 天内回复
-
maintenance
难度 2/5 1-3 小时 新手友好度 62/100
apex-dev-tools/apex-ls#603 ·
维护者通常 1 天内回复