Idle CU sweep cannot reclaim a computing unit whose execution is stuck in a non-terminal state
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 38/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- kubernetes, scala
- Domain
- backend, infrastructure
Research direction
Start with ComputingUnitHelpers.reconcileVanishedKubernetesUnits and the idle Kubernetes computing unit sweep, then trace how workflow_executions status and last_update_time are used. Compare the existing listing-triggered reconciliation with the proposed scheduled pod-state check. Done means a live pod with a stale non-terminal execution no longer keeps an idle computing unit unreclaimed.
Written by the indexing model from the issue text.
Description
Feature Summary
Follow-up to #6046.
Problem
The idle Kubernetes computing unit sweep treats a computing unit as busy whenever any of its
workflow_executions rows carries a non-terminal status code (status NOT IN (3, 4, 5)). That
test has no time bound, so an execution row left stuck in a non-terminal state keeps its computing unit off the sweep indefinitely, even though the unit is doing no work.
But the problem is, nothing else reclaims it either: ComputingUnitHelpers.reconcileVanishedKubernetesUnits only
runs when someone calls a listing endpoint, and it only checks whether the pod is already gone,
not whether the execution row is stuck somewhere. A live pod with a stuck row is missed on both paths.
maptoStatusCode also maps UNKNOWN (and TERMINATED) to -1, which is not in {3,4,5}. A row whose final status is -1 pins its unit off the sweep permanently. Fixing that means changing the codes amber writes, not this sweep's predicate.
Proposed Solution or Design
- Ignore a non-terminal execution row whose
last_update_timeis older than its own timeout. - Check execution status codes against actual pod state on a schedule, rather than only on a
listing request.
Either is a larger change than #6046 should carry, hence this follow-up.
Affected Area
No response
- Dominant language
- Scala
- Stars
- 316
- Forks
- 192
- Avg merge
- 4d 17h
- Merged PRs (30d)
- 141
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/texera
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/texera#8775 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
apache/texera#8756 · 4 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/texera#8700 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
apache/texera#8682 · 5 comments ·
Maintainers usually reply within 1 day
-
JSONL File Scan reads a JSON null as the text "null"Possibly taken @CaroFernando claimed this 4 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
apache/texera#8674 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
A-consensus C-question
Difficulty 1/5 Under an hour Newbie friendliness 74/100
ergoplatform/ergo#2624 ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
[Rust][Flaky Test] multiple_deadlines_fire_in_order asserts a wall-clock gap instead of firing orderOpenCI/CD ⚒️ Flaky-tests 🐦
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
valkey-io/valkey-glide#7255 ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
lichess-org/lila#21905 ·
Maintainers usually reply within 1 day