Rolling maintenance fails with ErrorInMaintenance because the cluster it disables is excluded as VM migration destination
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- java
- Domain
- backend, infrastructure
Research direction
Start with RollingMaintenanceManagerImpl.disableClusterIfEnabled() and startRollingMaintenanceHostInCluster(), then trace how DeploymentPlanningManagerImpl filters disabled clusters for VmWorkMigrateAway. Reproduce in a two-host, single-cluster zone with migrate.vm.across.clusters=false and inspect migration tests around rolling maintenance and destination selection. Done means VMs can evacuate to the remaining host, maintenance completes, and an aborted run does not leave the cluster disabled.
Written by the indexing model from the issue text.
Description
problem
startRollingMaintenance disables the cluster itself (RollingMaintenanceManagerImpl.disableClusterIfEnabled() sets the allocation state to Disabled) before putting hosts into maintenance. The VMs on the host then cannot be migrated to the other host in the same cluster, because the deployment planner excludes Disabled clusters. The host ends up in ErrorInMaintenance after the maintenance timeout, and the cluster is left in Disabled state.
Management server log:
DEBUG [c.c.d.DeploymentPlanningManagerImpl] Adding clusters [1] of pod [1] to the void set because these clusters are in the Disabled state.
WARN [c.c.d.DeploymentPlanningManagerImpl] VM last host ID: 2 belongs to zone ID: 1 for which config - migrate.vm.across.clusters is false and storage migration would be needed for inter-cluster migration, therefore, adding all other clusters except ID: 1 from this zone to avoid list
WARN [c.c.v.ClusteredVirtualMachineManagerImpl] Unable to find destination for migrating the vm VM instance {"id":786,"instanceName":"i-2-786-VM","type":"User",...}
ERROR [c.c.v.VmWorkJobDispatcher] Unable to complete AsyncJob {"cmd":"com.cloud.vm.VmWorkMigrateAway",...}
InsufficientServerCapacityException: Unable to find a server to migrate to.Scope=interface com.cloud.dc.DataCenter; id=1
...
ERROR [c.c.r.RollingMaintenanceManagerImpl] Timeout: waited 1800000ms for host <uuid>(<host>) to be in Maintenance state, but after timeout it is in ErrorInMaintenance
at RollingMaintenanceManagerImpl.waitForHostInMaintenance(RollingMaintenanceManagerImpl.java:775)
at RollingMaintenanceManagerImpl.putHostIntoMaintenance(RollingMaintenanceManagerImpl.java:411)
at RollingMaintenanceManagerImpl.startRollingMaintenanceHostInCluster(RollingMaintenanceManagerImpl.java:327)
at RollingMaintenanceManagerImpl.startRollingMaintenance(RollingMaintenanceManagerImpl.java:210)
The same migrate-away failure occurs for the system VMs, the virtual router and the user VMs on the host.
Root cause (from code and logs):
startRollingMaintenancecallsdisableClusterIfEnabled()beforestartRollingMaintenanceHostInCluster().DeploymentPlanningManagerImplvoids Disabled clusters, so withmigrate.vm.across.clusters=false(or a single-cluster zone) no destination is left.attemptMaintain()then finds running VMs and no pending migration retries, and callssetHostIntoErrorInMaintenance().
versions
- CloudStack: main branch (4.22.1.0-SNAPSHOT)
- Hypervisor: KVM (Ubuntu 26.04)
- Zone with a single cluster of 2 hosts,
migrate.vm.across.clusters=false
The steps to reproduce the bug
- Have an Enabled cluster with 2 KVM hosts, with system VMs, a VR and user VMs running on the second host.
- Run
startRollingMaintenanceon the cluster (cmk:start rollingmaintenance clusterids=<cluster id>). - Wait for the first host to enter maintenance.
What to do about it?
Expected: VMs are migrated to the other host, the host reaches Maintenance, rolling maintenance continues, and the cluster is re-enabled at the end.
Actual: the host goes to ErrorInMaintenance after the timeout and the cluster stays Disabled.
Possible fixes:
- Allow migrations triggered by host maintenance (
VmWorkMigrateAway) to target hosts in a cluster disabled by rolling maintenance, or do not use theDisabledallocation state to block new deployments. - Always re-enable the clusters that rolling maintenance disabled when it aborts.
- Fail fast instead of waiting for the full timeout when all migrations have failed.
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.4k
- Avg merge
- 7d 4h
- Merged PRs (30d)
- 13
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/cloudstack
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/cloudstack#14719 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
apache/cloudstack#14715 ·
Maintainers usually reply within 1 day
-
resizeVolume of a running system VM's volume is accepted but the VM never sees the new sizePossibly taken @bhouse-nexthop claimed this 7 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/cloudstack#14645 · 1 comment ·
Maintainers usually reply within 1 day
-
VR reports a load balancer change as applied when haproxy.cfg cannot be replacedPossibly taken @bhouse-nexthop claimed this 7 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/cloudstack#14633 · 1 comment ·
Maintainers usually reply within 1 day
-
restartNetwork reports a ResourceUnavailableException as an unexpected error (530), unlike restartVPC (534)Possibly taken @bhouse-nexthop claimed this 7 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/cloudstack#14631 · 1 comment ·
Maintainers usually reply within 1 day
All issues in apache/cloudstack
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
SAP/ai-sdk-java#1206 ·
Maintainers usually reply within 1 day
-
enhancement good first issue insight-module
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
sdlc-pro/spring-lens#606 ·
Maintainers usually reply within 1 day
-
quality-assurance
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Paul-Austin-Oswego-CSC480-HCI521/gift-app#252 ·
Maintainers usually reply within 1 day
-
backlog kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
TencentBlueKing/bk-job#4627 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 63/100
openrocket/openrocket#3460 ·
Maintainers usually reply within 1 day