Can we improve merge performance?
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- python
- Domain
- data, performance
Research direction
No file or test is named. Start by locating the merger entry point and reproducing the slowdown with the finepdf English and nemotron-cc-1.0 medium datasets. Done means a measured improvement in merge performance without changing the resulting data.
Written by the indexing model from the issue text.
Description
Merger are running on very big data in single thread. This is slow. Can we improve?
Some datasets with issues:
finepdf english
nemotron-cc-1.0 medium
- Dominant language
- Python
- Stars
- 2
- Forks
- 2
- Avg merge
- 2h 42m
- Merged PRs (30d)
- 17
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from OpenEuroLLM/training-data-packer
-
Linter shall check annotation data integrityPossibly taken @mikemikimike claimed this 1 day ago. Openenhancement
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
OpenEuroLLM/training-data-packer#132 ·
Maintainers usually reply within 1 day
-
bug good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
OpenEuroLLM/training-data-packer#94 ·
Maintainers usually reply within 1 day
-
Calculate time and array sizePossibly taken @mrunesson claimed this today. Open
OpenEuroLLM/training-data-packer#140 · 1 assignee ·
Maintainers usually reply within 1 day
-
enhancement idea
Difficulty 4/5 3-5 days Newbie friendliness 25/100
OpenEuroLLM/training-data-packer#71 ·
Maintainers usually reply within 1 day
-
idea
Difficulty 4/5 3-5 days Newbie friendliness 48/100
OpenEuroLLM/training-data-packer#61 · 1 reaction ·
Maintainers usually reply within 1 day
All issues in OpenEuroLLM/training-data-packer
Similar issues
-
bug documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
awslabs/visual-asset-management-system#514 ·
Maintainers usually reply within 1 day
-
backend bug spec:REQ-053
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
nolte/kamerplanter#2547 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
EleutherAI/lm-evaluation-harness#4425 ·
Maintainers usually reply within 1 day
-
Gemini: respect PLANNING_DISABLED=1 in lifecycle context hooksPossibly taken @AmariAugustin claimed this today. Openbug good first issue help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 87/100
OthmanAdi/planning-with-files#329 · 1 comment ·
Maintainers usually reply within 1 day
-
wr.dynamodb.read_items returns [{}] for a single partition value that does not existPossibly taken @auraseb claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
aws/aws-sdk-pandas#3520 ·
Maintainers usually reply within 2 days