feature: type promotion and column reordering in Transaction V2 actions
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start by reading the Transaction V2 design and the related issues #7954, #8644, #8645, #8646, and #8658. The issue identifies two missing actions: type promotion that preserves field IDs while handling old data files, and column reordering. Done means designing and implementing support for both schema changes; no implementation files or tests are named in the issue.
Written by the indexing model from the issue text.
Description
Transaction V2 (#7954) can't express two schema changes that other table formats support (Iceberg has both):
- Type promotion (e.g. int32 → int64, float → double).
AlterFieldhas no type facet. Under non-reusable field ids (#8658) a cast isDropField+AddField+ data, so the field gets a new id and loses its indices and any id-based tracking. A widening that keeps the field id needs a design for reading data files written under the old type. - Column order.
AddFieldalways appends, and no action moves a field.
Found in a design review of the Transaction V2 stack (#7954, #8644, #8645, #8646).
- Dominant language
- Rust
- Stars
- 7.2k
- Forks
- 913
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 405
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lance-format/lance
-
bug: remove the "not stable yet" warning from DataReplacement commitsPossibly taken @lance-gatefixer claimed this today. Openbug
Difficulty 1/5 Under an hour Newbie friendliness 84/100
lance-format/lance#10433 ·
Maintainers usually reply within 3 days
-
feature performance
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
lance-format/lance#10137 ·
Maintainers usually reply within 3 days
-
Upgrade DataFusion for the unparser stack-overflow fixMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
lance-format/lance#8225 ·
Maintainers usually reply within 3 days
-
enhancement
Difficulty 1/5 Under an hour Newbie friendliness 84/100
lance-format/lance#6425 · 2 comments ·
Maintainers usually reply within 3 days
-
Expose compute_pq_distanceMay be free again A pull request for this issue was closed without being merged. Openenhancement
Difficulty 1/5 1-3 hours Newbie friendliness 62/100
lance-format/lance#5675 · 1 comment ·
Maintainers usually reply within 3 days
All issues in lance-format/lance
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
FalkorDB/FalkorDB#3597 · 1 comment ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
mozilla-services/syncstorage-rs#2715 ·
Maintainers usually reply within 1 day