Add support for Parquet Input Stream optimisations.
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- java
- Domain
- data-engineering, performance
Research direction
Start by reading the Parquet Java Input Stream Optimisations document and the Analytics Accelerator for S3 document linked in the issue. Then define the scope of the proposed parquet-io module around vectored reads, tail and small-file reads, page-index access, and sequential prefetching. Done would require an agreed implementation plan and support for the listed optimisations.
Written by the indexing model from the issue text.
Description
This issue tracks adding of a new module [parquet-io] with the goal of adding IO optimisations to the parquet-java repository.
The goal is to have a single place where the following optimisations can be implemented:
- Vectored Reads
- Reading the tail of a file in one request rather than multiple small requests (Avoid the parquet footer dance, multiple requests for the pageIndex)
- Small Parquet files are read in a single request
- Sequential prefetching
Parquet Java Input Stream Optimisations: Doc explains the features this will implement.
Analytics Accelerator for S3: Doc explains IO optimisations made in the analytics accelerator library.
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 28
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/parquet-java
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/parquet-java#3792 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/parquet-java#3767 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/parquet-java#3695 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/parquet-java#3667 ·
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/parquet-java#3587 ·
All issues in apache/parquet-java
Similar issues
-
bug untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
opensearch-project/ml-commons#5094 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
emitter:client:csharp feature
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
affects/8.10 affects/8.9 component/clients kind/bug likelihood/mid severity/mid
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Two open-case totals on one screen: the Programs tile says 15,858 and the nav badge says 15,868 Openbug frontend maui-pilot
Difficulty 2/5 1-3 hours Newbie friendliness 72/100