[Feature] Support Lance file format
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 45/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Domain
- data-engineering, databases
Research direction
Start by reading the existing Mosaic Rust FFI integration and lifecycle patterns, then compare the Java reference module at paimon-lance and inspect the existing file format interfaces. Done means an optional disabled-by-default CMake integration supports Lance reads and writes, required projections and row selection, row metadata, logical-type validation, round-trip tests, and Java/C++ compatibility fixtures.
Written by the indexing model from the issue text.
Description
Search before asking
- I searched in the issues and found nothing similar.
Motivation
Apache Paimon Java supports Lance as a data file format through the paimon-lance module. Paimon C++ currently cannot read or write tables configured with file.format=lance, which prevents Java and C++ engines from interoperating on those tables.
Solution
Add optional Lance file format support to Paimon C++, aligned with the Java implementation and Lance 0.39.0:
- register a Lance file format factory behind a disabled-by-default CMake option;
- support reading and writing Lance files through the existing file format interfaces;
- support column projection, precise row selection, row counts, and file row-id reporting;
- validate the same supported and unsupported logical types as Java;
- add C++ round-trip tests and Java/C++ compatibility fixtures in both directions.
The implementation should follow the existing Mosaic Rust FFI integration and lifecycle patterns.
Anything else?
Java reference module: https://github.com/apache/paimon/tree/master/paimon-lance
Are you willing to submit a PR?
- I am willing to submit a PR!
- Dominant language
- C++
- Stars
- 65
- Forks
- 29
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 78
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/paimon-cpp
-
enhancement
apache/paimon-cpp#381 · 1 assignee ·
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
apache/paimon-cpp#375 · 1 assignee ·
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 45/100
apache/paimon-cpp#361 · 1 assignee ·
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 45/100
apache/paimon-cpp#325 · 1 assignee ·
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 35/100
apache/paimon-cpp#319 · 1 reaction · 1 assignee ·
All issues in apache/paimon-cpp
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
objectionary/eo-graphs#74 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
enhancement
Difficulty 1/5 Under an hour Newbie friendliness 88/100
QuantStack/git2cpp#187 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100