pods get oom-killed when multiple heavy spark applications start at the same time
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 38/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- kubernetes, rust, spark
- Domain
- infrastructure
Research direction
Start with the secret-operator DaemonSet in the basic configuration and reproduce the issue by submitting a Spark application with 3,000 executors while observing the secret-operator pods. Verify whether the 128 MB memory limit causes OOM kills, then confirm that the selected higher limit prevents them during concurrent heavy applications.
Written by the indexing model from the issue text.
Description
Affected Stackable version
25.3.0
Current and expected behavior
when multiple spark applications get started at the same time, all of them having 300 executors, a lot of pvcs are submitted which need to be satisfied by the secret-operator. in basic configuration the secret operator container has 128mb of memory which seems to be not enough because in that case all the pods get oom-killed
Possible solution
Increase the memory limit of the secret-operator container from 128mb to 1gb
Additional context
@soenkeliebau like mentioned today, I'll try to get further details. I think this is reproducible by submitting a 3000 exec application and keeping an eye on the secret-operator ds
Environment
No response
Would you like to work on fixing this bug?
None
- Dominant language
- Rust
- Stars
- 13
- Forks
- 8
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 10
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from stackabletech/secret-operator
-
type/bug
Difficulty 4/5 3-5 days Newbie friendliness 45/100
stackabletech/secret-operator#754 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 48/100
stackabletech/secret-operator#753 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 72/100
stackabletech/secret-operator#720 · 1 comment ·
-
customer-request type/feature-improvement
stackabletech/secret-operator#630 · 7 comments · 1 assignee ·
-
stackabletech/secret-operator#620 · 2 comments · 1 assignee ·
All issues in stackabletech/secret-operator
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug core
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
JIT-compiled number -> Decimal conversion silently overflows instead of raising DECIMAL_OVERFLOW Openfuzz
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
ClickHouse/ClickHouse#122114 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
linebender/vello_svg#90 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100