I guess at one point or other we've all seen a black hole apiserver extension effect us in one way or other... What are the ways we should/can avoid such issues ? And, have we considered making the apiserver better at detecting/returning 500s in case of these sorts of issues?

(Am somewhat naive in this area so, please let me know if im missing something obvious)

What would you like to be added?

Yesterday, I saw an environment wherein gatekeeper was not up, and thus, the kubernetes controller manager was failing at doing basic tasks, like, it couldnt even manage to get a lease from etcd. When this happens,

• basic things like deployments dont work, or worse, provide users with incorrect info about the state of the cluster
• theres no way for the APIServer to really convey this back, bc many objects that the KCP manages just... never get updated.

I think in general, theres a problem we have w/ the APIServer where extensions and so on can bring it down. Ive seen this in passing from time to time.

Suggestion

So, what im wondering is, in the same way that we cordon kubelets when they dont speak up after 10 minutes.... Should we have some way of raising a red flag (i.e. maybe at the kubectl get level) that sais something like:

$> kubectl get deployment blah -o yaml 

status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2022-10-17T02:48:28Z"
    lastUpdateTime: "2022-10-17T02:48:28Z"
    message: Kubernetes Controller Manager has not updated this object in 6 hours
    reason: ZombieMode
    status: "True"
    type: Unknown

As far as implementation, I dont know the best way, but likely the APIServer, itself , could just have a similar health checking heartbeat thing like we do w/ kubelets, so that a periodic checkin from a healthy controller was provided, and in cases where this hasnt happened in a while, on read, the apiserver could append a condition to objects (or do some other metadata operation that made it totally obvious that an objct has become scale) .

Why is this needed?

Without this, users experience kubernetes control plane issues migght thing:

• etcd is corrupt
• their entire cluster is broken and requires an upgrade
• they need to reboot their nodes or pods to clear out stale client caches
• their CNI, or APIServer endpoint, has become split brained

or any other manner of very tricky and time consuming to troubleshoot issues. So, having a basic checkin expectation for controlplane components w/ a view that cascaded to commonly inspected objects (deployments, daemonsets, PVCs) for end users, would really make it easy to troubleshoot any issues related to apiserver black holes

ALTERNATIVE

MAybe, we can just make the APIServer more capable of responding intelligently when it is being toxically blocked by an APIServer webhook