Repository metrics
- Stars
- (4,267 stars)
- PR merge metrics
- (PR metrics pending)
Description
What steps did you take and what happened?
We are seeing occasional issues with nodes failing to join their cluster during upgrade due to a mismatch between the token the node expects, and the token that is actually refreshed by the kubeadm bootstrap provider. This is what it looks like in the cloud-init logs on an affected node:
I0411 04:40:48.255380 95726 token.go:250] [discovery] Retrying due to error: could not find a JWS signature in the cluster-info ConfigMap for token ID "org6mh"
I0411 04:40:53.240474 95726 token.go:230] [discovery] Waiting for the cluster-info ConfigMap to receive a JWS signature for token ID "org6mh"
At the same time, the kubeadm bootstrap provider is happily refreshing another token that is supposed to be used by this node:
I0411 04:15:14.583088 1 kubeadmconfig_controller.go:1396] "Bootstrap data secret for KubeadmConfig already exists, updating" controller="kubeadmconfig" controllerGroup="bootstrap.cluster.x-k8s.io" controllerKind="KubeadmConfig" KubeadmConfig="bm-bm08h-cluster-capi/cp-fz8kq" namespace="bm-bm08h-cluster-capi" name="cp-fz8kq" reconcileID="2c347912-eb6e-4a33-9793-4b9a4107bec9" Machine="bm-bm08h-cluster-capi/cp-fz8kq" resourceVersion="100138" KubeadmControlPlane="bm-bm08h-cluster-capi/cp" Cluster="bm-bm08h-cluster-capi/bm-bm08h-cluster" Secret="bm-bm08h-cluster-capi/cp-fz8kq"
I0411 04:20:28.270187 1 kubeadmconfig_controller.go:425] "Refreshing token until the infrastructure has a chance to consume it" controller="kubeadmconfig" controllerGroup="bootstrap.cluster.x-k8s.io" controllerKind="KubeadmConfig" KubeadmConfig="bm-bm08h-cluster-capi/cp-fz8kq" namespace="bm-bm08h-cluster-capi" name="cp-fz8kq" reconcileID="ec15f26b-f4a5-40ed-aec5-87ae303a261c" Machine="bm-bm08h-cluster-capi/cp-fz8kq" resourceVersion="100285" KubeadmControlPlane="bm-bm08h-cluster-capi/cp" Cluster="bm-bm08h-cluster-capi/bm-bm08h-cluster" Secret="kube-system/bootstrap-token-v3a7ub" oldExpiration="2025-04-11T04:30:02Z" newExpiration="2025-04-11T04:35:28Z"
Note org6mh vs v3a7ub. If we check the cluster-info ConfigMap, it indeed has the signature for the token that is refreshed (v3a7ub).
I have not been able to find any mentions of the "wrong" token in the logs from CABPK. It only seem to mention the token when actually refreshing it.
Context:
- Plain cluster (no ClusterClass)
- Happens during k8s upgrade
- Happens for both control plane and worker nodes
What did you expect to happen?
There shouldn't be a mismatch ever. The node should always try to join using the same token that the kubeadm bootstrap provider is refreshing.
Cluster API version
This has been happening at least since March and CAPI v1.9.7. We are now on CAPI v1.10.1.
Kubernetes version
v1.32.2 and v1.33.0 at least
Anything else you would like to add?
I am trying to gather more logs and resources and will update this issue with them. It may take a few days though, since it doesn't happen all the time.
We first though we were hitting this: https://github.com/kubernetes-sigs/cluster-api/pull/11520 But after bumping CAPI to a version where it was fixed, we are still seeing the issue.
Label(s) to be applied
/kind bug One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.