Address CNI installation race condition

What happened?

The issue is described in https://github.com/kubernetes-sigs/kubespray/issues/10928 however it was closed with a workaround that does not really resolve the underlying issue. In particular that PR does not change the default behaviour so scale.yml will still deploy broken nodes.

I think there are a few things to sort out to get to the bottom of this.

1. This says "make sure /opt/cni/bin exists". But if it is only meant to create the directories, it should not have recurse: true. (I vaguely recall very old versions of Ansible might have needed recurse: true to create parent directories, i.e. /opt/cni in this case?But modern versions of Ansible take care of that automatically when state: directory ).
2. However the bigger question is why is Ansible touching /opt/cni/bin at all. The Calico daemonset uses a hostPath volumeMount of /opt/cni/bin, and will create that directory if it is missing: https://github.com/kubernetes-sigs/kubespray/blob/master/roles/network_plugin/calico/templates/calico-node.yml.j2#L448 The comment says this is "Used to install CNI". But then ansible extracts a tarball on top of that location afterwards? https://github.com/kubernetes-sigs/kubespray/blob/v2.26.0/roles/network_plugin/cni/tasks/main.yml#L13 Seems messy. Is the CNI installed by the Daemonset via hostPath, or tarball via Ansible, or both?
3. Based on grep hostPath -A 1 -r kubespray/roles/network_plugin | grep /opt/cni/bin it looks like most or all of the CNI plugins are installed via hostPath so I'm not sure how specific this is to Calico, or could affect other CNIs too.

Maybe someone who knows more about the CNI installation could help indicate how to solve this.

What did you expect to happen?

https://github.com/kubernetes-sigs/kubespray/issues/10928 would be fixed.

How can we reproduce it (as minimally and precisely as possible)?

See https://github.com/kubernetes-sigs/kubespray/issues/10928

OS

Almalinux 8

Version of Ansible

N/A

Version of Python

N/A

Version of Kubespray (commit)

v2.22.2

Network plugin used

calico

Full inventory with variables

N/A

Command used to invoke ansible

See https://github.com/kubernetes-sigs/kubespray/issues/10928

Output of ansible run

See https://github.com/kubernetes-sigs/kubespray/issues/10928

Anything else we need to know

No response