CoreDNS pods running on the same Node after initial cluster creation

After a cluster with multiple control plane nodes is created all CoreDNS replicas are running on the same Node. Ideally they should be distributed across multiple nodes

Details

The kubeadm bootstrap provider is using kubeadm to deploy CoreDNS. It creates the CoreDNS deployment with preferredDuringSchedulingIgnoredDuringExecution. Because at the time of CoreDNS deployment there is only one Node all CoreDNS pods are scheduled on the same Node. There is no re-scheduling once additional CP nodes are added.

Related kubeadm issues, let's take a look!

• https://github.com/kubernetes/kubeadm/issues/1657
• https://github.com/kubernetes/kubeadm/issues/1931

Some idea on how to solve this

1. Change preferredDuringSchedulingIgnoredDuringExecution to requiredDuringSchedulingIgnoredDuringExecution for multi-CP control planes (also change it back during scale down to 1 replica)
   1. we should check if replica is always the same number (e.g. 2)
2. Delete CoreDNS Pods as soon as additional CP replicas join