CKS: cluster on a VPC tier fails when the VPC has the Firewall service (#12706)

Read the CKS paths named in the issue: setupKubernetesClusterVpcTierRules, the scale, add/remove worker paths, and setupKubernetesEtcdNetworkRules; compare them with setupKubernetesClusterIsolatedNetworkRules and createFirewallRules. Check the linked pull request #14670 for work already under way. Done means VPC tiers using the VPC Firewall service get ingress rules after LB/PF setup and keep them in sync during scale and node changes, while other VPC providers remain unchanged.

problem

Since #12706 (4.23.0.0) a VPC offering can include the Firewall service for public IPs. In such
a VPC the VPC virtual router drops all ingress traffic to a public IP that has port forwarding,
load balancing or static NAT rules, unless a firewall rule allows it: the VR is booted with
vpc_firewall_enabled=true and configure.py creates a FIREWALL_<ip> chain ending in DROP
for every such IP (_ensure_vpc_firewall_chains).

The Kubernetes Service does not know about this. For a cluster on a VPC tier it

1. acquires a new public IP for the cluster during the create job
   (acquireVpcTierKubernetesPublicIp, stored as cluster detail PUBLIC_IP_ID),
2. creates network ACL items for TCP 6443 and 22 on the tier (createVpcTierAclRules),
3. creates a load balancer (or port forwarding) rule for 6443 and SSH port forwarding rules
   2222+ on that IP (setupKubernetesClusterVpcTierRules),

but no firewall rule. Ingress firewall rules are only created for isolated networks
(setupKubernetesClusterIsolatedNetworkRules → createFirewallRules). The same
"VPC means no firewall" assumption (network.getVpcId() == null) is in

• KubernetesClusterScaleWorker.scaleKubernetesClusterVpcTierRules (scale),
• KubernetesClusterAddWorker.validateAndSetupNode / revertNetworkRules (add node),
• KubernetesClusterRemoveWorker.removeNodePortForwardingRules (remove node),
• KubernetesClusterStartWorker.setupKubernetesEtcdNetworkRules (etcd nodes).

Two things then fail on the cluster public IP:

• The management server cannot reach the nodes (SSH 2222+) and the API (6443).
• The control node cannot reach its own API: kubeadm-config.yaml has controlPlaneEndpoint
  and advertiseAddress set to the cluster public IP, so kubeadm init checks the API through
  the VR (hairpin). The first kubeadm init times out after writing the static pod manifests.
  deploy-kube-system then retries forever and every retry fails in preflight (ports in use,
  manifests exist, /var/lib/etcd not empty). No kubeconfig is ever written.

The cluster stays in Starting until cloud.kubernetes.cluster.start.timeout (default 3600 s)
and goes to Error. Adding the firewall rules by hand later does not help, because the control
node is already broken. The IP is acquired inside the create job, so the rules cannot be added
in advance either.

Observed on 4.23.0.0 (KVM, VPC VR provider, VPC offering with Firewall):

• listFirewallRules ipaddressid=<cluster IP> is empty; PF 2222 → control:22, PF 2223 →
  worker:22, LB api-lb 6443 and the ACL items 22/6443 (0.0.0.0/0,::/0) exist and are Active.
• Management server log, every 20 s from one minute after the job started:
  Waiting for Kubernetes cluster ... control node VMs to be accessible.
• TCP connects from the management server to the cluster IP on 2222, 2223 and 6443 time out.
• After 3600 s the job fails with Failed to setup Kubernetes cluster : <name> is not in usable state as the system is unable to access control node VMs of the cluster.
• With firewall rules added 30 minutes into the job: the management server reaches 6443 at once
  (API has been successfully provisioned), but the ready-node check keeps failing
  (NumberFormatException: For input string: "0\nE1110 ... http://localhost:8080/api ... connection refused", kubectl on the control node has no kubeconfig). On the control node
  deploy-kube-system had restarted more than 4000 times with the preflight errors above. The job
  then fails with ... does not have desired number of nodes in ready state.
• With the same firewall rules created within seconds after CKS created the LB and PF rules
  (what the fix does): the cluster was Running in under three minutes, and a later scale-up
  also succeeded.
• Deleting the cluster releases the public IP, and that revokes the firewall rules on it.

versions

• Apache CloudStack 4.23.0.0; the code paths are unchanged on main (checked at a7be64af7a)
• KVM, advanced zone, VPC with the VPC virtual router as provider for all VPC services
• VPC offering with the Firewall service (provider VpcVirtualRouter), tier offering with LB and
  conserve mode, custom network ACL list
• Kubernetes version v1.35.4 (calico ISO), default SystemVM template for the nodes

The steps to reproduce the bug

1. Create a VPC offering that includes the Firewall service (provider VpcVirtualRouter) and
   enable it. Create a VPC from it.
2. Create a tier with a custom network ACL list (not default_deny, which CKS rejects).
3. Create a CKS cluster on that tier (1 control node, 1 worker is enough).
4. listFirewallRules ipaddressid=<cluster public IP> returns nothing. The cluster stays in
   Starting, the management server log repeats ... control node VMs to be accessible, and
   after cloud.kubernetes.cluster.start.timeout the cluster is in Error.

Expected: CKS opens TCP 6443 and the node SSH ports on the cluster public IP, as it does for
isolated networks, and keeps them in sync on scale and node add/remove.

What to do about it?

When the VPC virtual router provides the Firewall service of the VPC (the same check that sets
vpc_firewall_enabled on the VR), CKS should create the ingress firewall rules on VPC tiers
too, in the same places where it does for isolated networks, but after the load balancing
and port forwarding rules. Firewall rules created on a fresh VPC public IP before any LB/PF rule
break the IP association (separate issue #14669). Firewall rules on VPC public IPs are
stored with vpc_id and without network_id, so the per-node lookups must go by public IP
instead of findByNetworkIdAndPorts. On cluster deletion the rules go away with the release of
the cluster public IP.

A pull request with this change follows. It leaves VPCs without the Firewall service and the
NSX/Netris paths unchanged.

Related:

• #12706 introduced the Firewall service for VPC public IPs.
• #14669: firewall rules before LB/PF on a new VPC public IP prevent its association
  with the VR.
• #13970 / #14212: the source CIDR of the CKS firewall rules is hard-coded to 0.0.0.0/0. The
  fix uses the same default as isolated networks, so those issues then apply to VPC tiers too.
  Note that a restricted 6443 rule must include the tier CIDR, because the nodes reach the API
  through the public IP.
• The cloud controller manager (apache/cloudstack-kubernetes-provider) has the same gap for
  LoadBalancer services: it opens either a firewall rule or a network ACL item, depending on
  the services of the tier network, never both (cloudstack_loadbalancer.go). In a VPC with the
  Firewall service and a custom ACL one of the two layers is missing. Separate issue.