CKS: reconcile Kubernetes node capacity after live CPU or memory scaling

Start at KubernetesClusterScaleWorker.scaleKubernetesClusterOffering() and trace the existing VM upgrade workflow, CKS node handling, and control-plane SSH/kubectl path. Implement the focused reconciliation helper described in the issue, then add unit and integration coverage for guest resources, kubelet state, Node readiness, capacity, allocatable values, schedulability, and failure handling.

Context

PR #13226 removed the remaining CKS restriction that prevented changing node compute offerings while a KVM cluster was running. Its implementation calls the CloudStack VM upgrade workflow for each CKS VM, and its validation covered the KVM/libvirt domain definition.

Follow-up discussion confirmed that the PR scope ends at the KVM/hypervisor layer and that Kubernetes-specific steps may still be required.

The Kubernetes documentation explicitly states that, for CPU and/or memory updates, calling the kubelet allocatable-resources endpoint is not sufficient and the kubelet must be restarted so that resource capacity and allocatable are correct:

https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#monitoring-device-plugin-resources

Related upstream Kubernetes reports:

• https://github.com/kubernetes/kubernetes/issues/124650
• https://github.com/kubernetes/enhancements/issues/4609

Problem

KubernetesClusterScaleWorker.scaleKubernetesClusterOffering() currently performs, for each CKS VM:

userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), ...);

After a successful live KVM resize, CloudStack can report the new offering and libvirt can expose the additional vCPU/RAM to the VM, while Kubernetes still advertises the node's old values in:

.status.capacity.cpu
.status.capacity.memory
.status.allocatable.cpu
.status.allocatable.memory

That leaves three potentially different views of the same node:

1. CloudStack service offering / VM metadata;
2. resources visible in the guest OS;
3. resources advertised by the Kubernetes Node object and used by the scheduler.

The current workflow neither verifies the guest OS nor refreshes and verifies the Kubernetes view. A successful CKS scale job can therefore report success even though the Kubernetes scheduler cannot use the newly added resources.

Proposed behavior

For a running Cloud-managed CKS cluster, when a node offering change increases the CPU count and/or RAM, reconcile each Kubernetes node sequentially:

1. Read and record the node's current schedulability, Kubernetes capacity, and allocatable values.
2. If it was schedulable, run kubectl cordon <node>.
3. Live-resize the VM through the existing CloudStack VM upgrade workflow.
4. Verify inside the guest OS that the expected online CPU count and memory are visible.
5. Restart kubelet on that node.
6. Wait for systemctl is-active kubelet, Kubernetes Node Ready=True, and updated capacity / allocatable values.
7. Restore schedulability only if CKS cordoned the node; preserve a node that was already cordoned by the operator.
8. Continue with the next node only after the current node is healthy and verified.

This must be a rolling operation. Do not resize all nodes first and restart all kubelets afterwards.

Cordon versus drain

A kubelet service restart does not power off the VM and does not itself require workload eviction. For this live-resize path, cordon is the default safety mechanism: it blocks new scheduling while the node is being resized and while Kubernetes still has stale capacity, but leaves existing Pods running.

drain should not be added unconditionally because it evicts workloads, is subject to PodDisruptionBudgets, and materially changes the impact of a live operation. Drain remains appropriate for a workflow that will stop/reboot the VM or for a future explicit operator policy.

The Kubernetes node documentation describes cordon / unschedulable as a preparatory maintenance step that prevents new Pods without affecting existing Pods:

https://kubernetes.io/docs/concepts/architecture/nodes/#manual-node-administration

The drain documentation describes drain as the operation used before bringing down/deleting a machine:

https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/

When no extra kubelet restart is needed

• A Created cluster has no running node to reconcile.
• For a Stopped cluster, the normal VM start also starts kubelet, which discovers the new resources; verify after cluster start rather than adding a second restart.
• An offering change that modifies only CPU cap/speed or another non-capacity attribute, while CPU count and RAM remain unchanged, does not need a kubelet capacity refresh.
• Dedicated external etcd VMs are not Kubernetes Node objects and do not run kubelet in the CKS image. Verify guest resources, but do not run kubectl cordon or restart kubelet for those VMs.

Verification requirements

The scale job must not succeed based only on the CloudStack VM record or libvirt XML.

For worker and control-plane Kubernetes nodes:

• guest online CPU count equals the target offering's CPU count;
• guest total memory increased and is consistent with the target offering, allowing normal kernel/virtualization overhead;
• kubelet is active after restart;
• the Node is Ready=True;
• .status.capacity.cpu equals the online guest CPU count;
• .status.capacity.memory is consistent with guest total memory;
• .status.allocatable.cpu and/or .status.allocatable.memory increased when the corresponding capacity increased;
• the node returns to its original schedulability state.

Use Kubernetes resource-quantity parsing or structured JSON/JSONPath output. Do not compare formatted kubectl describe text.

Failure behavior

• If cordon fails, do not resize that node.
• If the VM resize succeeds but guest verification fails, fail the CKS scale operation with the node and observed/expected values.
• If kubelet restart or Kubernetes verification fails, leave the node cordoned when that is safer, report the exact recovery action, and do not proceed to the next node.
• Always best-effort restore a node that CKS itself cordoned when verification proves it is healthy.
• Never uncordon a node that was already unschedulable before this operation.
• A retry must recognize a VM that already has the target offering and continue the missing guest/Kubernetes reconciliation rather than treating it as fully complete.

Suggested implementation

Add a focused orchestration helper used by KubernetesClusterScaleWorker, for example KubernetesClusterNodeCapacityReconciler, responsible for:

• reading Node status through the control-plane SSH/kubectl path;
• resolving the target node's SSH endpoint for isolated, VPC, and direct-access networks;
• cordon/preserve-schedulability handling;
• guest CPU/RAM verification;
• kubelet restart;
• Ready/capacity/allocatable verification;
• bounded retries and actionable diagnostics.

The scale worker should compare the old and target offering before changing each VM and invoke this reconciler only for a running Kubernetes node whose CPU count or RAM changes.

Acceptance criteria

1. A live CPU increase on a running KVM CKS worker finishes only after guest CPU and Node capacity/allocatable reflect the increase.
2. A live RAM increase finishes only after guest memory and Node capacity/allocatable reflect the increase.
3. Kubelet is restarted exactly once per affected Kubernetes node during a successful attempt.
4. Nodes are processed sequentially and are cordoned during the inconsistent window.
5. A previously schedulable node is uncordoned after successful verification.
6. A previously cordoned node remains cordoned.
7. Existing Pods are not drained/evicted merely to restart kubelet.
8. Control-plane nodes are reconciled sequentially; external etcd-only VMs are not treated as Kubernetes Nodes.
9. Stopped/Created clusters do not receive an unnecessary kubelet restart.
10. Cap-only offering changes do not receive an unnecessary kubelet restart.
11. Failures show which of CloudStack, guest OS, kubelet, Node readiness, capacity, or allocatable verification failed.
12. Unit and integration tests verify the complete CloudStack → guest → Kubernetes state transition, not only libvirt XML.

Suggested labels

• component:cks
• component:kubernetes
• type:bug