Rolling maintenance fails with ErrorInMaintenance because the cluster it disables is excluded as VM migration destination

Open
#14,516 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
java

Research direction

Start with RollingMaintenanceManagerImpl.disableClusterIfEnabled() and startRollingMaintenanceHostInCluster(), then trace how DeploymentPlanningManagerImpl filters disabled clusters for VmWorkMigrateAway. Reproduce in a two-host, single-cluster zone with migrate.vm.across.clusters=false and inspect migration tests around rolling maintenance and destination selection. Done means VMs can evacuate to the remaining host, maintenance completes, and an aborted run does not leave the cluster disabled.

Written by the indexing model from the issue text.

Description

bug
problem

startRollingMaintenance disables the cluster itself (RollingMaintenanceManagerImpl.disableClusterIfEnabled() sets the allocation state to Disabled) before putting hosts into maintenance. The VMs on the host then cannot be migrated to the other host in the same cluster, because the deployment planner excludes Disabled clusters. The host ends up in ErrorInMaintenance after the maintenance timeout, and the cluster is left in Disabled state.

Management server log:

DEBUG [c.c.d.DeploymentPlanningManagerImpl] Adding clusters [1] of pod [1] to the void set because these clusters are in the Disabled state.
WARN  [c.c.d.DeploymentPlanningManagerImpl] VM last host ID: 2 belongs to zone ID: 1 for which config - migrate.vm.across.clusters is false and storage migration would be needed for inter-cluster migration, therefore, adding all other clusters except ID: 1 from this zone to avoid list
WARN  [c.c.v.ClusteredVirtualMachineManagerImpl] Unable to find destination for migrating the vm VM instance {"id":786,"instanceName":"i-2-786-VM","type":"User",...}
ERROR [c.c.v.VmWorkJobDispatcher] Unable to complete AsyncJob {"cmd":"com.cloud.vm.VmWorkMigrateAway",...}
      InsufficientServerCapacityException: Unable to find a server to migrate to.Scope=interface com.cloud.dc.DataCenter; id=1
...
ERROR [c.c.r.RollingMaintenanceManagerImpl] Timeout: waited 1800000ms for host <uuid>(<host>) to be in Maintenance state, but after timeout it is in ErrorInMaintenance
    at RollingMaintenanceManagerImpl.waitForHostInMaintenance(RollingMaintenanceManagerImpl.java:775)
    at RollingMaintenanceManagerImpl.putHostIntoMaintenance(RollingMaintenanceManagerImpl.java:411)
    at RollingMaintenanceManagerImpl.startRollingMaintenanceHostInCluster(RollingMaintenanceManagerImpl.java:327)
    at RollingMaintenanceManagerImpl.startRollingMaintenance(RollingMaintenanceManagerImpl.java:210)

The same migrate-away failure occurs for the system VMs, the virtual router and the user VMs on the host.

Root cause (from code and logs):

  • startRollingMaintenance calls disableClusterIfEnabled() before startRollingMaintenanceHostInCluster().
  • DeploymentPlanningManagerImpl voids Disabled clusters, so with migrate.vm.across.clusters=false (or a single-cluster zone) no destination is left.
  • attemptMaintain() then finds running VMs and no pending migration retries, and calls setHostIntoErrorInMaintenance().
versions
  • CloudStack: main branch (4.22.1.0-SNAPSHOT)
  • Hypervisor: KVM (Ubuntu 26.04)
  • Zone with a single cluster of 2 hosts, migrate.vm.across.clusters=false
The steps to reproduce the bug
  1. Have an Enabled cluster with 2 KVM hosts, with system VMs, a VR and user VMs running on the second host.
  2. Run startRollingMaintenance on the cluster (cmk: start rollingmaintenance clusterids=<cluster id>).
  3. Wait for the first host to enter maintenance.
What to do about it?

Expected: VMs are migrated to the other host, the host reaches Maintenance, rolling maintenance continues, and the cluster is re-enabled at the end.

Actual: the host goes to ErrorInMaintenance after the timeout and the cluster stays Disabled.

Possible fixes:

  • Allow migrations triggered by host maintenance (VmWorkMigrateAway) to target hosts in a cluster disabled by rolling maintenance, or do not use the Disabled allocation state to block new deployments.
  • Always re-enable the clusters that rolling maintenance disabled when it aborts.
  • Fail fast instead of waiting for the full timeout when all migrations have failed.
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
7d 4h
Merged PRs (30d)
13

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/cloudstack

All issues in apache/cloudstack

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.