[flink] Respect low filesystem rate limits in remove_orphan_files
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- java
- Domain
- data-engineering
Research direction
Locate the remove_orphan_files implementation and trace how the configured total filesystem rate is assigned across scan subtasks. Check the existing rate-limiting tests or add coverage for a total rate below the scan parallelism; done means fractional per-subtask rates preserve the requested aggregate best-effort limit.
Written by the indexing model from the issue text.
Description
Search before asking
- I searched in the issues and found nothing similar.
Fluss version
0.9.0 (latest release)
Please describe the bug 🐞
remove_orphan_files assigns each scan subtask a minimum filesystem operation rate of 1/s. When the scan parallelism exceeds the configured total rate, this increases the aggregate rate above the requested limit.
For example, a total rate of 3 operations/s with parallelism 8 results in an aggregate target of 8 operations/s.
Solution
Allow fractional per-subtask rates by dividing the configured total rate by the scan parallelism, while retaining the existing best-effort rate-limiting semantics.
Are you willing to submit a PR?
- I'm willing to submit a PR!
- Dominant language
- Java
- Stars
- 2.2k
- Forks
- 628
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 138
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/fluss
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
infinispan/infinispan#18150 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
opensearch-project/k-NN#3597 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100