CKS upgrade fails in air-gapped clusters when the new K8s version bumps the pause image (kubeadm upgrade health-check Job scheduled on worker without the image)

Abierto
  • ¿Algo no está bien?
#14,654 0 comentarios 2 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
bash, java, kubernetes
Área
devops, infrastructure

Línea de trabajo

Start with KubernetesClusterUpgradeWorker, upgradeKubernetesClusterNodes(), and upgrade-kubernetes.sh; trace how ISO images are imported and how the upgrade script’s output is handled. Reproduce the failure in an offline upgrade where the target pause image is missing on workers. Done means the health-check Job can run using images from the ISO and upgrade failures provide the script output in management-server.log.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
problem

Since kubeadm runs a CreateJob health check during upgrade apply/upgrade plan,
it creates a Job upgrade-health-check-* in kube-system using the pause image of the
TARGET version (registry.k8s.io/pause:3.10.2 for 1.36.5).

CKS upgrades node 0 (control) first. The upgrade script imports ISO images only on the
node it runs on, so at this point the workers still only have pause:3.10.1.
The control node has also just been drained (cordoned), so the Job pod is scheduled on a
worker, tries to pull from registry.k8s.io, fails (air-gapped), the Job hits its deadline,
and kubeadm upgrade apply fails. The fallback retry only ignores
CoreDNSUnsupportedPlugins, so it fails again. The upgrade fails after all retries.

kubeadm-plan.log

versions
COMPONENT NAME

Kubernetes Service (CKS) – KubernetesClusterUpgradeWorker / upgrade-kubernetes.sh

CLOUDSTACK VERSION

4.23

ENVIRONMENT
  • Upgrade: v1.35.9 -> v1.36.5 (offline ISO, no internet on nodes)
  • containerd 1.7.12, 1 control + 2 workers, Cilium CNI
OS
  • Ubuntu 24.04
The steps to reproduce the bug
STEPS TO REPRODUCE
  1. Deploy an offline CKS cluster on a version whose pause tag differs from the target
    (1.35.x -> pause 3.10.1)
  2. Register the 1.36.5 ISO (contains pause 3.10.2) and upgrade the cluster
  3. kubectl -n kube-system get events | grep upgrade-health-check
EXPECTED RESULTS

Upgrade succeeds using only images from the ISO.

ACTUAL RESULTS

Failed to pull image "registry.k8s.io/pause:3.10.2": ... lookup registry.k8s.io ... i/o timeout
Job was active longer than specified deadline
(control node left cordoned, still v1.35.9; worker has only pause:3.10.1)
kubeadm upgrade plan v1.36.5 reproduces it (hangs on "Job ... is not yet complete").

LOGS

1. Health-check Job events (one full cycle, the same pattern repeated for 8 Jobs between 56m and 46m)

$ kubectl -n kube-system get events --sort-by=.lastTimestamp | grep -iE 'upgrade-health|pause|pull'
56m  Normal   SuccessfulCreate  job/upgrade-health-check-1794277494890       Created pod: upgrade-health-check-1794277494890-vbh5b
56m  Normal   Scheduled         pod/upgrade-health-check-1794277494890-vbh5b Successfully assigned kube-system/upgrade-health-check-1794277494890-vbh5b to k8s-upgrade-node-1a1c33de33f
56m  Warning  Failed            pod/upgrade-health-check-1794277494890-vbh5b Failed to pull image "registry.k8s.io/pause:3.10.2": failed to pull and unpack image "registry.k8s.io/pause:3.10.2": failed to resolve reference "registry.k8s.io/pause:3.10.2": failed to do request: Head "https://registry.k8s.io/v2/pause/manifests/3.10.2": dial tcp: lookup registry.k8s.io on 127.0.0.53:53: read udp 127.0.0.1:45086->127.0.0.53:53: i/o timeout
55m  Warning  Failed            pod/upgrade-health-check-1794277494890-vbh5b Error: ErrImagePull
55m  Normal   BackOff           pod/upgrade-health-check-1794277494890-vbh5b Back-off pulling image "registry.k8s.io/pause:3.10.2"
55m  Warning  Failed            pod/upgrade-health-check-1794277494890-vbh5b Error: ImagePullBackOff
55m  Normal   SuccessfulDelete  job/upgrade-health-check-1794277494890       Deleted pod: upgrade-health-check-1794277494890-vbh5b
55m  Warning  DeadlineExceeded  job/upgrade-health-check-1794277494890       Job was active longer than specified deadline

Jobs created: 1794277494890, 1794277555081, 1794277651281, 1794277711462,
1794277807947, 1794277868124, 1794277969440, 1794278029604. All were scheduled on the worker
k8s-upgrade-node-1a1c33de33f and all failed the same way. 8 Jobs = 4 script runs
(upgrade retries) x 2 kubeadm upgrade apply attempts per run.

2. Reproduce with kubeadm upgrade plan (control node)

$ kubeadm upgrade plan v1.36.5 --v=5
[upgrade] Running cluster health checks
I1110 03:21:49.429624 health.go:182] Creating a Job "upgrade-health-check-1794280909429" in the namespace "kube-system"
I1110 03:21:49.449846 health.go:210] Job "upgrade-health-check-1794280909429" in the namespace "kube-system" is not yet complete, retrying
... (repeats every 1s until deadline)

3. Image state: control has the image, worker does not

# control node (images imported from 1.36.5 ISO)
$ kubeadm config images list --kubernetes-version v1.36.5 | grep pause
registry.k8s.io/pause:3.10.2
$ crictl images | grep pause
registry.k8s.io/pause   3.10.1   cd073f4c5f6a8   320kB
registry.k8s.io/pause   3.10.2   4a83b15d3ecfe   321kB

# worker node (not yet upgraded)
$ crictl images | grep pause
registry.k8s.io/pause   3.10.1   cd073f4c5f6a8   320kB

4. Cluster state after failure

$ kubectl get node
NAME                              STATUS                     ROLES           AGE   VERSION
k8s-upgrade-control-1a1c33cc713   Ready,SchedulingDisabled   control-plane   57m   v1.35.9
k8s-upgrade-node-1a1c33d569a      Ready                      <none>          56m   v1.35.9
k8s-upgrade-node-1a1c33de33f      Ready                      <none>          56m   v1.35.9

kubeadm binary on control is already v1.36.5, but static pods and kubelet are still v1.35.9
(the script exits at kubeadm upgrade apply).
Environment: containerd 1.7.12, imageRepository: registry.k8s.io,
sandbox_image = "registry.k8s.io/pause:3.10.2" (control).

WORKAROUND

Export pause:3.10.2 from the control node, import it on all workers, then retry the upgrade.

What to do about it?
SUGGESTED FIX
  1. Pre-stage phase: after attachIsoKubernetesVMs, import ISO images on ALL cluster VMs
    before upgrading node 0 (e.g. split image import out of upgrade-kubernetes.sh into a
    separate script/flag run on every node first).
  2. Log result.second() (script stdout/stderr) on failure in
    upgradeKubernetesClusterNodes(). It is currently discarded, which makes the
    failure invisible in management-server.log.
  3. Minor: upgrade-kubernetes.sh checks $# -lt 4 but its usage text and the Java caller use 5 args;
    PAUSE_IMAGE selection via ctr images ls | grep pause | sort | tail -1 is fragile.
    Use kubeadm config images list | grep pause instead.
Lenguaje dominante
Java
Estrellas
3.1k
Forks
1.4k
Merge medio
7 d 16 h
PR fusionados (30 d)
15

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/cloudstack

Todos los issues de apache/cloudstack

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.