VPC VR: static-NAT SNAT rule can precede the site-to-site VPN NAT exemption, so a static-NAT VM's VPN traffic leaves with its public IP

Start in systemvm/debian/opt/cloud/bin/configure.py at CsForwardingRules.processStaticNatRule and CsSite2SiteVpn.configure_iptables, then inspect CsNetfilter.py for front insertion and has_rule behavior. Reproduce the ordering-dependent case by enabling static NAT after the VPN connection; done means VPN-marked traffic retains its private source while internet-bound traffic still uses public SNAT.

problem

On a VPC virtual router, the per-VM static-NAT rule

-A POSTROUTING -o <public dev> -s <vm>/32 -j SNAT --to-source <public ip>

and the site-to-site VPN NAT exemption

-A POSTROUTING -o <public dev> -m mark --mark 0x525 -j ACCEPT

are both inserted at the front of the nat POSTROUTING chain, and whichever is created later ends up first. When static NAT is enabled on a VM after a site-to-site VPN connection already exists, the SNAT rule precedes the exemption and every packet that VM sends into the tunnel carries its public address instead of its private one. The remote side sees traffic from the public IP, its rules for the VPC's guest CIDR do not match, and that VM cannot reach the remote network while VMs without static NAT can. Nothing in the API or UI indicates a problem: the connection stays Connected.

Where the other end of the tunnel is does not matter (another VPC, another zone, another cloud, a customer gateway); the translation happens on the local VPC router before encryption.

Observed:

• Ping and TCP/22 from the static-NAT VM (A) to a VM behind the peer (B) fail. B's kernel receives no echo requests (/proc/net/snmp InEchos does not move).
• Temporarily allowing all ingress on B's tier makes the traffic arrive, and a listener on B reports the connection from A's static-NAT public address, not from the VPC's guest CIDR.
• Traffic initiated by B toward A is not affected (connection tracking keeps the reply un-translated), so the failure is one-directional and only for the VM with static NAT.
• The order of operations decides the outcome. Measured on the same setup:
  • static NAT enabled on A before the VPN gateway, customer gateway and connection are created: traffic crosses with private addresses;
  • static NAT disabled and re-enabled after the connection exists: ICMP and TCP/22 from A both fail;
  • the connection deleted and recreated with static NAT left in place: traffic crosses again.
• Replacing the static NAT with a port-forwarding rule also avoids it, since port forwarding only source-translates hairpin traffic to the VM's own public IP.

Analysis (4.22.1.1 sources):

• systemvm/debian/opt/cloud/bin/configure.py, CsForwardingRules.processStaticNatRule, lines 1559-1560 (4.23.0.0: line 1685):
  ["nat", "front", "-A POSTROUTING -o %s -s %s/32 -j SNAT --to-source %s"]
• configure.py, CsSite2SiteVpn.configure_iptables, line 1059 (4.23.0.0: line 1184):
  ["nat", "front", "-A POSTROUTING -t nat -o %s -m mark --mark 0x525 -j ACCEPT"]
• systemvm/debian/opt/cloud/bin/cs/CsNetfilter.py, lines 164-176: front becomes iptables -I <chain> <rule> (insert at position 1), and a rule that already exists is skipped (has_rule).

So on a fresh apply the databags are processed forwardingrules first and site2sitevpn second (configure.py lines 1591-1595), the exemption is inserted last and lands on top, and everything works. On an incremental apply only the new rule is inserted: enabling static NAT after the connection exists puts the SNAT rule at position 1 above the existing exemption, so the VM's VPN-bound packets are translated before the exemption can match them. The order then persists until the connection is deleted and recreated (measured) or the router is rebuilt (expected from the fresh-apply order).

The VPN mark itself is set correctly (mangle FORWARD -s <vpc cidr> -d <peer cidr> -j MARK --set-xmark 0x525), so the exemption matches whenever it is reached. The VPC-wide source-NAT rule is appended rather than front-inserted and is not affected; only per-VM static NAT is.

versions

• Apache CloudStack 4.22.1.1, KVM, advanced zone, VPC with "Default VPC offering" (NAT mode, VPC virtual router), DefaultIsolatedNetworkOfferingForVpcNetworks tiers, IKEv2 site-to-site connections between two VPC virtual routers.
• The same rules and insertion logic are present in 4.23.0.0 (line numbers above).

The steps to reproduce the bug

1. Two VPCs in one zone with non-overlapping CIDRs, e.g. 10.90.0.0/22 (A) and 10.91.0.0/22 (B), one tier each, one VM each. Tier ACLs on both sides allow ICMP and TCP/22 from the peer VPC CIDR. (A second VPC is only the simplest peer; any site-to-site endpoint reproduces it.)
2. createVpnGateway on each VPC, a createVpnCustomerGateway on each side describing the other (peer gateway IP, peer VPC CIDR, IKEv2), and createVpnConnection on each side. Send traffic once; both connections reach Connected.
3. associateIpAddress on VPC A and enableStaticNat it to VM A.
4. From VM A, ping VM B's private address and open TCP/22 to it.

Expected: traffic between VM A and VM B crosses the tunnel with private source and destination addresses, as it does for a VM without static NAT. Static NAT should apply to internet-bound traffic only.

Actual: both fail; with the peer tier temporarily open, VM B sees the connection arriving from VM A's public address.

What to do about it?

Make the static-NAT rule itself skip VPN-marked traffic, so chain order stops mattering:

-A POSTROUTING -o <dev> -s <vm>/32 -m mark ! --mark 0x525 -j SNAT --to-source <public ip>

Alternatives: re-assert the VPN exemption at position 1 whenever forwarding rules are (re)applied, or stop inserting the per-VM SNAT rule at the front. The first option is the smallest change, and it matches what a Neutron router does for floating IPs, which are exempted from source translation by an IPsec policy match.

Remote-access VPN was not checked.