Allow the serving container to set restartPolicy and restartPolicyRules (KEP-5307), behind a feature flag
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 38/100
Research direction
The proposal identifies the feature flag, field mask, validation, schema, and revision reconciler work, with unit, table, and end-to-end tests planned. Start by locating the existing serving-container API validation and feature-flag patterns, then trace how the revision reconciler observes Deployment readiness and container termination. Done means the gated fields are copied to the Deployment and eligible terminated pods are replaced, with the specified status behavior covered by tests.
Written by the indexing model from the issue text.
Description
/kind feature
/area API
/area autoscale
Describe the feature
Let the serving container of a Knative Service set the container-level restartPolicy and restartPolicyRules added by KEP-5307, behind a new feature flag kubernetes.containerspec-restartpolicy (disabled by default). When the serving container stops and Kubernetes won't restart it, Knative should remove that pod so the Deployment replaces it.
Background
This was discussed in #13041. In the WG call on 2022-07-06 the WG agreed to expose restartPolicy behind a feature gate, but the issue was closed because a Deployment only allows the pod-level restartPolicy: Always (kubernetes/kubernetes#24725). #14185 was closed for the same reason.
KEP-5307 changes that. The restart policy can now be set per container, and the pod-level policy stays Always. The feature gate ContainerRestartRules is beta and on by default since Kubernetes 1.35, which is Knative's current minimum version. A Deployment accepts it: on Kubernetes 1.35 the API server accepts and stores a Deployment whose pod template has restartPolicy: Always and a container with restartPolicy: Never and restart rules.
Use case
Some apps need to end their pod when they finish, and restart in place when they fail:
- Exit 0 means the app is done with this pod, for example because it held per-pod state that must not be reused. The pod should be replaced by a new one.
- A non-zero exit is a crash and should restart the container in place, with the kubelet's normal back-off, as today.
With KEP-5307 this is either:
containers:
- image: example.com/app
restartPolicy: OnFailure
or, to restart only on some exit codes:
containers:
- image: example.com/app
restartPolicy: Never
restartPolicyRules:
- action: Restart
exitCodes:
operator: NotIn
values: [0]
The rules also address the concern raised in #13157 about telling transient failures from permanent ones: the user states which exit codes are permanent.
The problem: a stopped serving container leaves the pod Running
Exposing the field alone isn't enough, because of the queue-proxy. I tested on kind with Kubernetes 1.35, using a Deployment shaped like a Revision's: a serving container and a long-running second container with a TCP readiness probe on the app's port.
Step 1 – serving container exits 0, its policy says don't restart
→ container state: terminated, restarts=0
Step 2 – the other container (queue-proxy) keeps running
→ pod phase stays Running, Ready=False
Step 3 – ReplicaSet: desired=1 current=1 ready=0
✗ the pod is never replaced; the replica is lost until someone deletes it
The same test with exit code 3 restarts the container in place and the pod becomes Ready again. ✓
Today's revision reconciler wouldn't report this either. It reads only LastTerminationState, which is empty for a container that was never restarted, and only when no replica is available.
Proposal
- API, behind
kubernetes.containerspec-restartpolicy(disabled by default). With the flag on, the serving container may setrestartPolicyandrestartPolicyRules, copied unchanged to the Deployment. Both are still refused on init containers (restartPolicy: Alwaysthere makes a native sidecar, a separate topic) and on non-serving containers (see question 2). Kubernetes validates the values and rules; Knative doesn't copy its limits. - Pod cleanup in the revision reconciler. When the serving container sets a restart policy and the Deployment has fewer ready replicas than desired, list the Revision's pods. Delete any pod, not already being deleted, whose serving container is terminated and won't be restarted under its policy and rules. Emit an Event on the Revision. The Deployment then creates a replacement. The controller already has
deleteon pods, and is already enqueued when the Deployment's ready count drops. - Status. Report a non-zero exit that won't be restarted on the Revision's
ContainerHealthycondition, as is done today for crash loops. Exit 0 isn't marked unhealthy.
Revisions that don't set a restart policy don't change and pay no extra cost.
Open questions
- No back-off for deleted pods. An app that exits 0 right at startup would be deleted and replaced in a fast loop, without the kubelet's CrashLoopBackOff. Proposal: delete only pods that had been Ready at least once. A serving container that stops before its pod was ever Ready is reported on the Revision as a failure, the same way crash loops are today.
- Sidecars. Should non-serving containers get these fields too? A stopped sidecar would also leave the pod not ready. Proposal: serving container only for now.
- Draining. When the app ends with exit 0, it has to finish its in-flight requests before it exits. The queue-proxy has no way to drain it first. This should be documented.
- Kubernetes GA.
ContainerRestartRulesis beta and on by default, and sig-node has deferred GA while waiting for production feedback (kubernetes/enhancements#5307). Is a disabled-by-default Knative flag on a beta Kubernetes field acceptable, or should it wait for GA? If a cluster turns the gate off, the API server drops the fields and the container falls back to today'sAlways.
Plan
- PR 1: the feature flag, field mask, validation and schema, with unit tests.
- PR 2: pod cleanup and status in the revision reconciler, with table tests and an e2e test that runs with the flag on.
- Docs: a knative/docs entry on the feature-flags page.
I'm happy to do the work and can join a Serving WG meeting to discuss it.
We already run this behind a flag in our own build on Kubernetes 1.35 in production.
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 1.2k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 51
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from knative/serving
-
progress-deadline annotation accepts "0s" although validation says it must be positivePossibly taken @BahaaMattar claimed this today. Openarea/API kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
envFrom validation errors have no entry index, and their paths are garbled when several entries are invalidPossibly taken @BahaaMattar claimed this today. Openarea/API kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
auto:flaky
Difficulty 4/5 3-5 days Newbie friendliness 35/100
knative/serving#16789 · 1 comment ·
Maintainers usually reply within 1 day
-
auto:flaky
Difficulty 3/5 1-2 days Newbie friendliness 55/100
knative/serving#16788 · 1 comment ·
Maintainers usually reply within 1 day
-
auto:flaky
Difficulty 4/5 3-5 days Newbie friendliness 25/100
knative/serving#16787 · 1 comment ·
Maintainers usually reply within 1 day
Similar issues
-
[Plugin] calendarOpen
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
MUKE-coder/grit#228 ·
-
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 87/100
Chemaclass/agnostic-ai#3681 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
OSS TEST
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
seaweedfs/seaweed-block#268 ·
Maintainers usually reply within 1 day