Spark CR deletion leaves orphaned pods
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- kubernetes, rust
- Domain
- distributed-systems, infrastructure
Research direction
Start by tracing SparkApplication deletion through the CR's deletionTimestamp and the driver and executor pod owner references. Investigate whether a finalizer can trigger driver deletion while preserving the existing cleanup behavior, then verify that deleting the CR removes the driver and cascaded executor pods without leaving them orphaned.
Written by the indexing model from the issue text.
Description
Affected Stackable version
26.3.0
Affected Apache Spark-on-Kubernetes version
all
Current and expected behavior
Before 26.3
CR held owner reference onto driver and driver onto executer which lead to deletion of pods when deleting the CR. It was convenient since reinstalling the SparkApp requires to delete the old CR.
After 26.3
CR doesn't have owner reference on driver pod, driver still owns the executer. Means, deleting the CR now leaves driver and executer untouched and thus run uncontrolled until manually deletion. The operator auto deletes the driver pod once it reaches "terminating" phase which might not happen during uninstalls e.g.
Proposal
Deleting a CR purposefully should lead to the product being uninstalled. Maybe finalizer on the SparkApplication can help. We could on deletionTimestamp delete the driver pod which then cascades to the executer due to ownerRef. This would preserve the behaviour that if driver reaches terminating, it gets cleaned up by the operator and thus its executor.
Possible solution
No response
Additional context
No response
Environment
No response
Would you like to work on fixing this bug?
None
- Dominant language
- Rust
- Stars
- 72
- Forks
- 4
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 15
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from stackabletech/spark-k8s-operator
-
type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
stackabletech/spark-k8s-operator#774 · 1 comment ·
-
type/bug
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
type/bug
Difficulty 3/5 1-2 days Newbie friendliness 55/100
-
type/bug
Difficulty 3/5 1-2 days Newbie friendliness 52/100
stackabletech/spark-k8s-operator#730 · 6 comments ·
All issues in stackabletech/spark-k8s-operator
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug core
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
JIT-compiled number -> Decimal conversion silently overflows instead of raising DECIMAL_OVERFLOW Openfuzz
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
ClickHouse/ClickHouse#122114 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
linebender/vello_svg#90 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100