Generated partition columns resolve to null when column mapping is enabled
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- java
- Domain
- data-engineering
Research direction
Start by reading getSerializedPartitionValue and its partitionFieldNames branch, along with the review discussion on #973 that identified the suspected lookup issue. Trace how generated partition columns are resolved when column mapping is enabled for both Kernel and Standalone Delta sources. Done means the example column yr resolves to its partition value instead of null in both sources, with and without #973.
Written by the indexing model from the issue text.
Description
Follow-up from review discussion on #973 (reported by @vaibhavk1992, not blocking that PR).
With delta.columnMapping.mode=name enabled, a generated partition column still comes out null on both the Kernel and Standalone Delta sources, with and without #973. The same table without column mapping resolves correctly.
Example: yr INT GENERATED ALWAYS AS (YEAR(ts)), partitioned by yr.
Suspected cause (from the review): the partitionFieldNames branch of getSerializedPartitionValue looks values up by the generated column's logical name, and the generated column isn't in the internal schema, so there's no storageName to use.
This is outside the scope of #972, so tracked here separately.
- Dominant language
- Java
- Stars
- 1.3k
- Forks
- 220
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 12
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/incubator-xtable
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/incubator-xtable#977 ·
Maintainers usually reply within 3 days
-
0.5.0
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
apache/incubator-xtable#921 · 2 comments ·
Maintainers usually reply within 3 days
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/incubator-xtable#893 ·
Maintainers usually reply within 3 days
-
Missing org.openjdk.jol from imported dependencies in xtable-utilities makes it to fail with NoClassDefFoundErrorPossibly taken A pull request linked to this issue is open or already merged. Open0.5.0 bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/incubator-xtable#736 · 8 comments ·
Maintainers usually reply within 3 days
-
0.5.0 documentation
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
apache/incubator-xtable#717 · 4 comments ·
Maintainers usually reply within 3 days
All issues in apache/incubator-xtable
Similar issues
-
agent:working bug follow-up lane:auto
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Sanne/incus-spawn#1765 · 2 comments ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
checkstyle/checkstyle#22260 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
strimzi/strimzi-kafka-operator#13328 · 1 comment · 3 reactions ·
Maintainers usually reply within 1 day