AvroReadSupport should automatically detect file list encoding
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- java
- Domain
- data-engineering
Research direction
Look at AvroReadSupport in the parquet-avro module to understand how list structures are currently handled. The detection logic must examine the Parquet schema's repeated group patterns to distinguish three-level and two-level lists. Testing will involve reading existing Parquet files with both list encodings to ensure automatic detection works without the configuration property.
Written by the indexing model from the issue text.
Description
Describe the enhancement requested
parquet-avro currently contains opt-in support for reading and writing both "three-level" lists:
required group mylist (LIST) {
repeated group list {
required int32 element;
}
}
and "two level"/"legacy"/"old" lists:
required group mylist (LIST) {
repeated int32 array;
}
The boolean Configuration property parquet.avro.write-old-list-structure controls which structure should be written, but readers have to specify the parquet.avro.write-old-list-structure property, too, if they're either (a) supplying a custom projection, or (b) reading a Parquet file that doesn't contain the parquet.avro.schema key in its file metadata.
Ideally the reader should be able to automatically detect both types of list encoding, so that we can keep the Configuration property as a writer-side implementation detail.
Component(s)
Avro
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 6d 16h
- Merged PRs (30d)
- 36
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/parquet-java
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/parquet-java#3792 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/parquet-java#3767 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/parquet-java#3695 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/parquet-java#3667 ·
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/parquet-java#3587 ·
All issues in apache/parquet-java
Similar issues
-
area/plugin
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
kestra-io/plugin-kestra#190 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
google-ai-edge/LiteRT-LM#3739 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
integra-team-red/meet-map#249 ·
-
[Studio][Bug] Cancelled create-user dialog keeps the password and admin switch for the next attempt Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
apache/rocketmq-dashboard#5064 ·