Discussion around storing SUM as a optional metadata field.
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Active
- Domain
- data-engineering
Research direction
Start by reviewing the existing Parquet metadata fields and optional bloom-filter handling described in the issue. Define how optional SUM metadata would be represented and how overflow and SUM/AVG pre-aggregation should work. Done means the format design and its trade-offs are agreed.
Written by the indexing model from the issue text.
Description
Describe the enhancement requested
Currently parquet only stores fields for predicate pushdown for scan pruning, but we can also keep sum in metadata field, while this won't help in predicate pushdown, but it can act as a pre-aggregation for sum and avg queries.
While this may be challenging and bit complex because we would need to handle sum/overflow etc. But this can be optional just like bloom filters and would be helpful for many usecases. And shouldn't have much of a overhead due to it being a stateless operation.
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 509
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 2
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/parquet-format
-
Type: enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/parquet-format#626 · 2 reactions ·
-
Variant shredding spec ambiguously refers to both SQL NULL and Variant null as "null"May be free again @nssalian claimed this 180 days ago, and no pull request is open. OpenType: enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/parquet-format#519 · 4 comments ·
-
[Format] Encoding spec incorrect for dictionary fallbackPossibly taken @d33bs claimed this 49 days ago. OpenPriority: Critical Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
apache/parquet-format#404 · 4 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 38/100
apache/parquet-format#627 · 5 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
apache/parquet-format#622 ·
All issues in apache/parquet-format
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
posit-dev/pointblank#442 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
openwatersio/slackwater-database#274 · 1 comment ·
Maintainers usually reply within 1 day
-
Triage 🩺
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
[BUG] flatten_keys(inplace=True) deletes the leaves at the root of the tensordictPossibly taken @peterdsharpe claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
pytorch/tensordict#2300 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
PolicyEngine/policyengine-uk#2296 ·
Maintainers usually reply within 1 day