design: pending timeout for pods that never schedule, and image pull failures on the infra rail
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 15/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
Research direction
Start in internal/executor/reconcile.go around lines 49-87, where Pending and the image-pull reasons are settled, then read pod_terminate.go:315 and stale_queued_reap.go:194, which defer on Pending. Read ADR 0051 and ADR 0053 for the intended infra rail and pending bound. This issue is a design proposal that depends on the infra settle path from the eviction issue, so agree on the design with maintainers before coding. Done means an unschedulable pod is failed after the configured timeout and image pull errors retry before settling.
Written by the indexing model from the issue text.
Description
Found in the 2026-10-09 performance and reliability re-audit (findings R-08 and R-09). Both reachable with default settings.
Pods that never schedule stay queued forever
internal/executor/reconcile.go:87 treats Pending as non-terminal unless a container waits on an unrecoverable reason, and an unscheduled pod has no container status. pod_terminate.go:315 and stale_queued_reap.go:194 treat Pending as a slow start and defer. activeDeadlineSeconds counts from status.startTime, which an unscheduled pod never gets, and nothing reads PodScheduled=False. A typo in node_selector, a missing toleration, a request larger than any node or a PVC that never binds leaves the task queued indefinitely, holding its pool slot and max_active_tasks. ADR 0053 asks for a bound here. Throwaway test on main 024196b9: an Unschedulable pod is still unsettled after 6 hours.
Image pull failures spend user retries
reconcile.go:49-50, 64-84 settle ErrImagePull and ImagePullBackOff as a task failure on the first observation. A registry 429 or 503 or a short network blip therefore fails the task and consumes retries (0 by default), although ADR 0051 puts bad images on the infra rail. Throwaway test: settled=true kind=failed on the first sweep.
Proposal
- A setting such as
executor.pending_timeout(default 10 to 15 minutes): a pod stillUnschedulablepast it is deleted and the task failed with a reason that names scheduling (infra re-place for capacity reasons if the operator prefers waiting). - Settle
ErrImagePullandImagePullBackOffas infra after a bounded number of back-offs, keepInvalidImageNameterminal. - Both need the infra settle path in the reconciler that the eviction issue introduces.
- Dominant language
- Go
- Stars
- 46
- Forks
- 1
- Avg merge
- 13h 15m
- Merged PRs (30d)
- 642
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dexadata/dexaflow
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
area/docs documentation priority/p3
Difficulty 1/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/ci enhancement priority/p3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/docs documentation priority/p2
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
area/docs documentation priority/p2
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
All issues in dexadata/dexaflow
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
nebari-dev/nebi#630 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
router-for-me/CLIProxyAPI#7084 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
speakeasy-api/openapi#284 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
yetone/magpie#4095 · 1 comment ·
Maintainers usually reply within 1 day