Is big Decimal in Parquet big endian?
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Stale
- Domain
- data-engineering
Research direction
Start with the Parquet data-pages encodings documentation linked in the issue and verify the byte order and encoding rules for BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY decimal values. Done means documenting whether big decimals use big-endian or little-endian bytes and when any encoding or byte swapping occurs.
Written by the indexing model from the issue text.
Description
Describe the usage question you have. Please include as many useful details as possible.
It's not clear from description https://parquet.apache.org/docs/file-format/data-pages/encodings/ how are big Decimals (precision >= 18) placed. As I could understand they can be encoded as BYTE_ARRAY or FIXED_LEN_BYTE_ARRAY. But there're two ways to place "the bytes contained in the array" from memory: big endian and little endian. I.e. they are in LE in int128 in memory. Could I place the bytes into FIXED_LEN_BYTE_ARRAY as is?
It's also not clean if there's some encoding over these decimal's bytes, should the data be swapped to correct endianness before of after the encoding?
Component(s)
No response
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 6d 16h
- Merged PRs (30d)
- 36
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/parquet-java
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/parquet-java#3792 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/parquet-java#3767 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/parquet-java#3695 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/parquet-java#3667 ·
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/parquet-java#3587 ·
All issues in apache/parquet-java
Similar issues
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
inu-appcenter/memorIN-backend#288 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
frontend maui-pilot pilot-ask question
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
area/plugin
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
kestra-io/plugin-kestra#190 ·