VXLAN persistent networks create bridges on hosts that never ran a VM on them, but don't clean them up
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 74/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- java
- Domain
- infrastructure, networking
Research direction
Start with NetworkOrchestrator.cleanupPersistentnNetworkResources() and networkMeetsPersistenceCriteria(), then compare them with DefaultHostListener.setupPersistentNetwork() and hostAboutToBeRemoved(). Reproduce the VXLAN persistent-network deletion flow on three or more hosts and inspect management-server.log and /sys/class/net/brvx-*; done means cleanup is sent and bridges and VXLAN interfaces disappear from every host.
Written by the indexing model from the issue text.
Description
problem
With a persistent network offering, CloudStack builds the network's bridge on every host in the zone. But when the network is deleted it only cleans the bridge up on hosts that actually ran a VM on it. On every other host, the bridge and its VXLAN interface stay behind forever.
Investigation
There is a difference between the filters when creating a persistent network, and removing a persistent network.
Creation:
DefaultHostListener.setupPersistentNetwork() creates the bridges on hostConnect/hostEnabled for all networks from getAllPersistentNetworksFromZone() - no isolation-method filter.
Removal:
NetworkOrchestrator.cleanupPersistentnNetworkResources() is gated by networkMeetsPersistenceCriteria(), which requires the broadcast URI scheme to be Vlan, so for vxlan:// networks CleanupPersistentNetworkResourceCommand is never sent. (hostAboutToBeRemoved() sends the same cleanup with no scheme check either - the VLAN-only gate exists only on the network-delete path.)
Hosts that ran a VM on the network are cleaned up via the normal VM-lifecycle teardown, so the leak only affects uninvolved hosts.
versions
4.22.1.1, KVM, advanced zone with VXLAN isolation, persistent network offerings.
The steps to reproduce the bug
- Advanced zone, KVM, VXLAN isolation, 3+ hosts.
- Create a VPC with two tiers on a persistent offering.
- Start VMs so they land on only some hosts. (Restarting
cloudstack-agenton an uninvolved host also creates the bridges there via hostConnect.) - Delete the tiers, then the VPC.
ls -d /sys/class/net/brvx-*on each host.
Expected behaviour:
The bridge and VXLAN interface are removed from every host in the zone.
Actual behaviour:
The VNI is still present on hosts that never ran a VM on the network; CleanupPersistentNetworkResourceCommand never appears in management-server.log during the delete.
What to do about it?
Accept Vxlan alongside Vlan in networkMeetsPersistenceCriteria(), or drop the scheme check to match the setup path.
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.4k
- Avg merge
- 6d 20h
- Merged PRs (30d)
- 27
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/cloudstack
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 90/100
apache/cloudstack#14222 ·
-
bug component:kubernetes
Difficulty 1/5 Under an hour Newbie friendliness 88/100
apache/cloudstack#14180 ·
-
bug component:projects component:UI
Difficulty 1/5 Under an hour Newbie friendliness 88/100
apache/cloudstack#14070 · 5 comments ·
-
component:backup
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/cloudstack#14013 ·
-
KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 Openbug component:ceph
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/cloudstack#13989 · 3 comments ·
All issues in apache/cloudstack
Similar issues
-
area/plugin
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
kestra-io/plugin-kestra#190 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
google-ai-edge/LiteRT-LM#3739 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
integra-team-red/meet-map#249 ·
-
[Studio][Bug] Cancelled create-user dialog keeps the password and admin switch for the next attempt Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
apache/rocketmq-dashboard#5064 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
wso2/dpdp-accelerator#287 ·