warm pools: pods that never become ready count as idle forever, and a crashing warm image is recreated every 30 s

Open
  • Is something wrong?
#1,622 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
28/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes
Domain
backend, infrastructure

Research direction

Start in internal/executor/warmpool_k8s.go around line 57, where only Succeeded and Failed are treated as terminal, then check how warmpool.go (452-468) counts idle pods and how internal/executor/warmpod.go creates them. Run TestAudit2PendingWarmPodCountsAsIdleForever to reproduce the stuck-Pending case. Done means a pod that never registers is not counted as idle and is replaced after a deadline, and repeated start failures back off per pool with a metric and event.

Written by the indexing model from the issue text.

Description

area/executor bug go priority/p3

Found in the 2026-10-09 performance and reliability re-audit (R-28, confirmed by two reviewers, first half proven by a test). Only reachable with warm pools on.

Problem

  • internal/executor/warmpool_k8s.go:57 treats only Succeeded and Failed as terminal, so a warm pod stuck Pending (unschedulable, ImagePullBackOff) or Terminating on a lost node counts as idle (warmpool.go:452-468). min_idle looks satisfied, the pool and tenant caps are charged, and every task silently goes to a dedicated pod. Warm pods have no activeDeadlineSeconds and no readiness signal (internal/executor/warmpod.go). TestAudit2PendingWarmPodCountsAsIdleForever: a 6 h old ImagePullBackOff pod with min_idle 1 triggers no new create.
  • A warm image that crashes on start is deleted and recreated on every 30 s reconcile, with no backoff.

Proposal

  1. Count a warm pod as idle only after its worker registered; a pod not registered within a deadline is deleted and replaced.
  2. Exponential backoff per pool after consecutive start failures, with a metric and an event naming the reason.
Dominant language
Go
Stars
46
Forks
1
Avg merge
11h 20m
Merged PRs (30d)
636

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dexadata/dexaflow

All issues in dexadata/dexaflow

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.