In `DataFramePartitioner.DataFrameFunctions.repartitionByRecordCount` use max instead of average to decide about partitoing
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- scala, spark
- Domain
- data-engineering, performance
Research direction
Start in DataFramePartitioner.DataFrameFunctions.repartitionByRecordCount and trace how record counts per partition are aggregated and compared with the limit. Run the existing test suite and verify that the maximum count, rather than the average, controls whether repartitioning occurs. Done when partitions over the limit trigger repartitioning while below-limit behavior remains unchanged.
Written by the indexing model from the issue text.
Description
Background
In DataFramePartitioner.DataFrameFunctions.repartitionByRecordCount the limit is compared against the average of records per partition. That can be under the limit while some partitions are still over the limit.
Feature
Change the method to decide about portioning based on the max of record counts per partition if to do or not the repartitioing.
Or make this parametrized or a dedicated method.
- Dominant language
- Scala
- Stars
- 9
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from AbsaOSS/spark-partition-sizing
-
bug
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 22/100
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
All issues in AbsaOSS/spark-partition-sizing
Similar issues
-
`IcebergWriteExec` and `CometIcebergWriteExec` are marked with `!` (missing input) in EXPLAINPossibly taken @Visorgood claimed this today. Openbug requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
apache/datafusion-comet#7622 · 2 comments ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
lichess-org/lila#22506 · 1 reaction ·
Maintainers usually reply within 1 day
-
15-minutes 5-minutes algorithm any-language beginner-friendly c cpp csharp easy go good first issue hacktoberfest help wanted java javascript kotlin python rust swift typescript
Difficulty 1/5 Under an hour Newbie friendliness 68/100
Huerte/algo-solutions-python#260 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
cpp-best-practices/project_validator#245 ·
Maintainers usually reply within 1 day
-
python 🐍 Task 🔧
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
valkey-io/valkey-glide#7709 ·
Maintainers usually reply within 1 day