Allow the serving container to set restartPolicy and restartPolicyRules (KEP-5307), behind a feature flag

Open
#16,750 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
38/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes
Domain
api, backend

Research direction

The proposal identifies the feature flag, field mask, validation, schema, and revision reconciler work, with unit, table, and end-to-end tests planned. Start by locating the existing serving-container API validation and feature-flag patterns, then trace how the revision reconciler observes Deployment readiness and container termination. Done means the gated fields are copied to the Deployment and eligible terminated pods are replaced, with the specified status behavior covered by tests.

Written by the indexing model from the issue text.

Description

area/API area/autoscale kind/feature

/kind feature
/area API
/area autoscale

Describe the feature

Let the serving container of a Knative Service set the container-level restartPolicy and restartPolicyRules added by KEP-5307, behind a new feature flag kubernetes.containerspec-restartpolicy (disabled by default). When the serving container stops and Kubernetes won't restart it, Knative should remove that pod so the Deployment replaces it.

Background

This was discussed in #13041. In the WG call on 2022-07-06 the WG agreed to expose restartPolicy behind a feature gate, but the issue was closed because a Deployment only allows the pod-level restartPolicy: Always (kubernetes/kubernetes#24725). #14185 was closed for the same reason.

KEP-5307 changes that. The restart policy can now be set per container, and the pod-level policy stays Always. The feature gate ContainerRestartRules is beta and on by default since Kubernetes 1.35, which is Knative's current minimum version. A Deployment accepts it: on Kubernetes 1.35 the API server accepts and stores a Deployment whose pod template has restartPolicy: Always and a container with restartPolicy: Never and restart rules.

Use case

Some apps need to end their pod when they finish, and restart in place when they fail:

  • Exit 0 means the app is done with this pod, for example because it held per-pod state that must not be reused. The pod should be replaced by a new one.
  • A non-zero exit is a crash and should restart the container in place, with the kubelet's normal back-off, as today.

With KEP-5307 this is either:

containers:
- image: example.com/app
  restartPolicy: OnFailure

or, to restart only on some exit codes:

containers:
- image: example.com/app
  restartPolicy: Never
  restartPolicyRules:
  - action: Restart
    exitCodes:
      operator: NotIn
      values: [0]

The rules also address the concern raised in #13157 about telling transient failures from permanent ones: the user states which exit codes are permanent.

The problem: a stopped serving container leaves the pod Running

Exposing the field alone isn't enough, because of the queue-proxy. I tested on kind with Kubernetes 1.35, using a Deployment shaped like a Revision's: a serving container and a long-running second container with a TCP readiness probe on the app's port.

Step 1 – serving container exits 0, its policy says don't restart
         → container state: terminated, restarts=0
Step 2 – the other container (queue-proxy) keeps running
         → pod phase stays Running, Ready=False
Step 3 – ReplicaSet: desired=1 current=1 ready=0
         ✗ the pod is never replaced; the replica is lost until someone deletes it

The same test with exit code 3 restarts the container in place and the pod becomes Ready again. ✓

Today's revision reconciler wouldn't report this either. It reads only LastTerminationState, which is empty for a container that was never restarted, and only when no replica is available.

Proposal
  1. API, behind kubernetes.containerspec-restartpolicy (disabled by default). With the flag on, the serving container may set restartPolicy and restartPolicyRules, copied unchanged to the Deployment. Both are still refused on init containers (restartPolicy: Always there makes a native sidecar, a separate topic) and on non-serving containers (see question 2). Kubernetes validates the values and rules; Knative doesn't copy its limits.
  2. Pod cleanup in the revision reconciler. When the serving container sets a restart policy and the Deployment has fewer ready replicas than desired, list the Revision's pods. Delete any pod, not already being deleted, whose serving container is terminated and won't be restarted under its policy and rules. Emit an Event on the Revision. The Deployment then creates a replacement. The controller already has delete on pods, and is already enqueued when the Deployment's ready count drops.
  3. Status. Report a non-zero exit that won't be restarted on the Revision's ContainerHealthy condition, as is done today for crash loops. Exit 0 isn't marked unhealthy.

Revisions that don't set a restart policy don't change and pay no extra cost.

Open questions
  1. No back-off for deleted pods. An app that exits 0 right at startup would be deleted and replaced in a fast loop, without the kubelet's CrashLoopBackOff. Proposal: delete only pods that had been Ready at least once. A serving container that stops before its pod was ever Ready is reported on the Revision as a failure, the same way crash loops are today.
  2. Sidecars. Should non-serving containers get these fields too? A stopped sidecar would also leave the pod not ready. Proposal: serving container only for now.
  3. Draining. When the app ends with exit 0, it has to finish its in-flight requests before it exits. The queue-proxy has no way to drain it first. This should be documented.
  4. Kubernetes GA. ContainerRestartRules is beta and on by default, and sig-node has deferred GA while waiting for production feedback (kubernetes/enhancements#5307). Is a disabled-by-default Knative flag on a beta Kubernetes field acceptable, or should it wait for GA? If a cluster turns the gate off, the API server drops the fields and the container falls back to today's Always.
Plan
  • PR 1: the feature flag, field mask, validation and schema, with unit tests.
  • PR 2: pod cleanup and status in the revision reconciler, with table tests and an e2e test that runs with the flag on.
  • Docs: a knative/docs entry on the feature-flags page.

I'm happy to do the work and can join a Serving WG meeting to discuss it.

We already run this behind a flag in our own build on Kubernetes 1.35 in production.

Dominant language
Go
Stars
6.1k
Forks
1.2k
Avg merge
1d 15h
Merged PRs (30d)
51

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from knative/serving

All issues in knative/serving

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.