CKS upgrade fails in air-gapped clusters when the new K8s version bumps the pause image (kubeadm upgrade health-check Job scheduled on worker without the image)

Start with KubernetesClusterUpgradeWorker, upgradeKubernetesClusterNodes(), and upgrade-kubernetes.sh; trace how ISO images are imported and how the upgrade script’s output is handled. Reproduce the failure in an offline upgrade where the target pause image is missing on workers. Done means the health-check Job can run using images from the ISO and upgrade failures provide the script output in management-server.log.

problem

Since kubeadm runs a CreateJob health check during upgrade apply/upgrade plan,
it creates a Job upgrade-health-check-* in kube-system using the pause image of the
TARGET version (registry.k8s.io/pause:3.10.2 for 1.36.5).

CKS upgrades node 0 (control) first. The upgrade script imports ISO images only on the
node it runs on, so at this point the workers still only have pause:3.10.1.
The control node has also just been drained (cordoned), so the Job pod is scheduled on a
worker, tries to pull from registry.k8s.io, fails (air-gapped), the Job hits its deadline,
and kubeadm upgrade apply fails. The fallback retry only ignores
CoreDNSUnsupportedPlugins, so it fails again. The upgrade fails after all retries.

kubeadm-plan.log

versions

COMPONENT NAME

Kubernetes Service (CKS) – KubernetesClusterUpgradeWorker / upgrade-kubernetes.sh

CLOUDSTACK VERSION

4.23

ENVIRONMENT

• Upgrade: v1.35.9 -> v1.36.5 (offline ISO, no internet on nodes)
• containerd 1.7.12, 1 control + 2 workers, Cilium CNI

OS

• Ubuntu 24.04

The steps to reproduce the bug

STEPS TO REPRODUCE

1. Deploy an offline CKS cluster on a version whose pause tag differs from the target
   (1.35.x -> pause 3.10.1)
2. Register the 1.36.5 ISO (contains pause 3.10.2) and upgrade the cluster
3. kubectl -n kube-system get events | grep upgrade-health-check

EXPECTED RESULTS

Upgrade succeeds using only images from the ISO.

ACTUAL RESULTS

Failed to pull image "registry.k8s.io/pause:3.10.2": ... lookup registry.k8s.io ... i/o timeout
Job was active longer than specified deadline
(control node left cordoned, still v1.35.9; worker has only pause:3.10.1)
kubeadm upgrade plan v1.36.5 reproduces it (hangs on "Job ... is not yet complete").

LOGS

1. Health-check Job events (one full cycle, the same pattern repeated for 8 Jobs between 56m and 46m)

$ kubectl -n kube-system get events --sort-by=.lastTimestamp | grep -iE 'upgrade-health|pause|pull'
56m  Normal   SuccessfulCreate  job/upgrade-health-check-1794277494890       Created pod: upgrade-health-check-1794277494890-vbh5b
56m  Normal   Scheduled         pod/upgrade-health-check-1794277494890-vbh5b Successfully assigned kube-system/upgrade-health-check-1794277494890-vbh5b to k8s-upgrade-node-1a1c33de33f
56m  Warning  Failed            pod/upgrade-health-check-1794277494890-vbh5b Failed to pull image "registry.k8s.io/pause:3.10.2": failed to pull and unpack image "registry.k8s.io/pause:3.10.2": failed to resolve reference "registry.k8s.io/pause:3.10.2": failed to do request: Head "https://registry.k8s.io/v2/pause/manifests/3.10.2": dial tcp: lookup registry.k8s.io on 127.0.0.53:53: read udp 127.0.0.1:45086->127.0.0.53:53: i/o timeout
55m  Warning  Failed            pod/upgrade-health-check-1794277494890-vbh5b Error: ErrImagePull
55m  Normal   BackOff           pod/upgrade-health-check-1794277494890-vbh5b Back-off pulling image "registry.k8s.io/pause:3.10.2"
55m  Warning  Failed            pod/upgrade-health-check-1794277494890-vbh5b Error: ImagePullBackOff
55m  Normal   SuccessfulDelete  job/upgrade-health-check-1794277494890       Deleted pod: upgrade-health-check-1794277494890-vbh5b
55m  Warning  DeadlineExceeded  job/upgrade-health-check-1794277494890       Job was active longer than specified deadline

Jobs created: 1794277494890, 1794277555081, 1794277651281, 1794277711462,
1794277807947, 1794277868124, 1794277969440, 1794278029604. All were scheduled on the worker
k8s-upgrade-node-1a1c33de33f and all failed the same way. 8 Jobs = 4 script runs
(upgrade retries) x 2 kubeadm upgrade apply attempts per run.

2. Reproduce with kubeadm upgrade plan (control node)

$ kubeadm upgrade plan v1.36.5 --v=5
[upgrade] Running cluster health checks
I1110 03:21:49.429624 health.go:182] Creating a Job "upgrade-health-check-1794280909429" in the namespace "kube-system"
I1110 03:21:49.449846 health.go:210] Job "upgrade-health-check-1794280909429" in the namespace "kube-system" is not yet complete, retrying
... (repeats every 1s until deadline)

3. Image state: control has the image, worker does not

# control node (images imported from 1.36.5 ISO)
$ kubeadm config images list --kubernetes-version v1.36.5 | grep pause
registry.k8s.io/pause:3.10.2
$ crictl images | grep pause
registry.k8s.io/pause   3.10.1   cd073f4c5f6a8   320kB
registry.k8s.io/pause   3.10.2   4a83b15d3ecfe   321kB

# worker node (not yet upgraded)
$ crictl images | grep pause
registry.k8s.io/pause   3.10.1   cd073f4c5f6a8   320kB

4. Cluster state after failure

$ kubectl get node
NAME                              STATUS                     ROLES           AGE   VERSION
k8s-upgrade-control-1a1c33cc713   Ready,SchedulingDisabled   control-plane   57m   v1.35.9
k8s-upgrade-node-1a1c33d569a      Ready                      <none>          56m   v1.35.9
k8s-upgrade-node-1a1c33de33f      Ready                      <none>          56m   v1.35.9

kubeadm binary on control is already v1.36.5, but static pods and kubelet are still v1.35.9
(the script exits at kubeadm upgrade apply).
Environment: containerd 1.7.12, imageRepository: registry.k8s.io,
sandbox_image = "registry.k8s.io/pause:3.10.2" (control).

WORKAROUND

Export pause:3.10.2 from the control node, import it on all workers, then retry the upgrade.

What to do about it?

SUGGESTED FIX

1. Pre-stage phase: after attachIsoKubernetesVMs, import ISO images on ALL cluster VMs
   before upgrading node 0 (e.g. split image import out of upgrade-kubernetes.sh into a
   separate script/flag run on every node first).
2. Log result.second() (script stdout/stderr) on failure in
   upgradeKubernetesClusterNodes(). It is currently discarded, which makes the
   failure invisible in management-server.log.
3. Minor: upgrade-kubernetes.sh checks $# -lt 4 but its usage text and the Java caller use 5 args;
   PAUSE_IMAGE selection via ctr images ls | grep pause | sort | tail -1 is fragile.
   Use kubeadm config images list | grep pause instead.