Include FAPEC compressor support to Parquet?

Open

#3,067 opened on Nov 18, 2024 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 5/5
Estimated time: Over a week
Newbie friendliness: 25/100

Issue type: Feature
Clarity: Needs clarification
Activity status: Stale
Tech stack: java
Domain: data

Research direction

The issue names no files, tests, or entry points. Start by reviewing how Parquet codec support is structured and the FAPEC proof of concept, then assess the licensing and binary-only constraints for encoding and decoding. Done would require a decided compatibility path and a concrete implementation scope.

Written by the indexing model from the issue text.

Description

Type: enhancement

Describe the enhancement requested

FAPEC is a high-performance data compression algorithm with many options, based on efficient entropy coding and including several pre-processing algorithms for time series, images, text, floats, etc.
It's already available for some formats like HDF5 and FITS. We're now investigating the possibility to include FAPEC as a new codec option for Parquet, which (we think) should provide better compression ratios (than the currently available codecs) mainly for integers and floats/doubles. We're now working on a proof-of-concept to evaluate how much would it actually improve.
If the outcome reveals a "significant enough" improvement, will it be possible to include this new option?
Note that FAPEC is a commercial product, so a valid license would be needed to generate Parquet files with this, and the compression library would have to be used in binary form for the adequate platform. (reading Parquet files compressed with FAPEC would always be free). Perhaps this would be a blocking issue...?

Component(s)

No response

Dominant language: Java
Stars: 3.1k
Forks: 1.6k
Avg merge: 6d 16h
Merged PRs (30d): 36

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

chore: aspectjweaver(EPL-2.0)와 AGPL-3.0 호환성 정리 Open

documentation

Difficulty 2/5 1-3 hours Newbie friendliness 65/100

inu-appcenter/memorIN-backend#288 · Opened Sep 24, 2026
Vulnerability Nessus 12085 - Apache Tomcat Default Files (with killbill/killbill:0.24.21) Open

Difficulty 2/5 1-3 hours Newbie friendliness 65/100

killbill/killbill#2314 · 1 comment · Opened Sep 24, 2026
Work list, Cases and Compliance overlap, and nothing says which one to use Open

frontend maui-pilot pilot-ask question

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

Taleef7/workwell#698 · Opened Sep 24, 2026
[Bug][UI] Replace the status of the button in the Sessions Tab Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

Netcracker/qubership-integration-platform#918 · Opened Sep 24, 2026
executions.Query — startDate and timeRange filters are sent with inverted comparison operators Open

area/plugin

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

kestra-io/plugin-kestra#190 · Opened Sep 24, 2026

More Java issues