Rolling maintenance fails with ErrorInMaintenance because the cluster it disables is excluded as VM migration destination

未关闭
  • 哪里不对？
#14,516 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
42/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
java
领域
backend, infrastructure

调研方向

Start with RollingMaintenanceManagerImpl.disableClusterIfEnabled() and startRollingMaintenanceHostInCluster(), then trace how DeploymentPlanningManagerImpl filters disabled clusters for VmWorkMigrateAway. Reproduce in a two-host, single-cluster zone with migrate.vm.across.clusters=false and inspect migration tests around rolling maintenance and destination selection. Done means VMs can evacuate to the remaining host, maintenance completes, and an aborted run does not leave the cluster disabled.

由索引模型根据 Issue 内容生成。

描述

bug
problem

startRollingMaintenance disables the cluster itself (RollingMaintenanceManagerImpl.disableClusterIfEnabled() sets the allocation state to Disabled) before putting hosts into maintenance. The VMs on the host then cannot be migrated to the other host in the same cluster, because the deployment planner excludes Disabled clusters. The host ends up in ErrorInMaintenance after the maintenance timeout, and the cluster is left in Disabled state.

Management server log:

DEBUG [c.c.d.DeploymentPlanningManagerImpl] Adding clusters [1] of pod [1] to the void set because these clusters are in the Disabled state.
WARN  [c.c.d.DeploymentPlanningManagerImpl] VM last host ID: 2 belongs to zone ID: 1 for which config - migrate.vm.across.clusters is false and storage migration would be needed for inter-cluster migration, therefore, adding all other clusters except ID: 1 from this zone to avoid list
WARN  [c.c.v.ClusteredVirtualMachineManagerImpl] Unable to find destination for migrating the vm VM instance {"id":786,"instanceName":"i-2-786-VM","type":"User",...}
ERROR [c.c.v.VmWorkJobDispatcher] Unable to complete AsyncJob {"cmd":"com.cloud.vm.VmWorkMigrateAway",...}
      InsufficientServerCapacityException: Unable to find a server to migrate to.Scope=interface com.cloud.dc.DataCenter; id=1
...
ERROR [c.c.r.RollingMaintenanceManagerImpl] Timeout: waited 1800000ms for host <uuid>(<host>) to be in Maintenance state, but after timeout it is in ErrorInMaintenance
    at RollingMaintenanceManagerImpl.waitForHostInMaintenance(RollingMaintenanceManagerImpl.java:775)
    at RollingMaintenanceManagerImpl.putHostIntoMaintenance(RollingMaintenanceManagerImpl.java:411)
    at RollingMaintenanceManagerImpl.startRollingMaintenanceHostInCluster(RollingMaintenanceManagerImpl.java:327)
    at RollingMaintenanceManagerImpl.startRollingMaintenance(RollingMaintenanceManagerImpl.java:210)

The same migrate-away failure occurs for the system VMs, the virtual router and the user VMs on the host.

Root cause (from code and logs):

  • startRollingMaintenance calls disableClusterIfEnabled() before startRollingMaintenanceHostInCluster().
  • DeploymentPlanningManagerImpl voids Disabled clusters, so with migrate.vm.across.clusters=false (or a single-cluster zone) no destination is left.
  • attemptMaintain() then finds running VMs and no pending migration retries, and calls setHostIntoErrorInMaintenance().
versions
  • CloudStack: main branch (4.22.1.0-SNAPSHOT)
  • Hypervisor: KVM (Ubuntu 26.04)
  • Zone with a single cluster of 2 hosts, migrate.vm.across.clusters=false
The steps to reproduce the bug
  1. Have an Enabled cluster with 2 KVM hosts, with system VMs, a VR and user VMs running on the second host.
  2. Run startRollingMaintenance on the cluster (cmk: start rollingmaintenance clusterids=<cluster id>).
  3. Wait for the first host to enter maintenance.
What to do about it?

Expected: VMs are migrated to the other host, the host reaches Maintenance, rolling maintenance continues, and the cluster is re-enabled at the end.

Actual: the host goes to ErrorInMaintenance after the timeout and the cluster stays Disabled.

Possible fixes:

  • Allow migrations triggered by host maintenance (VmWorkMigrateAway) to target hosts in a cluster disabled by rolling maintenance, or do not use the Disabled allocation state to block new deployments.
  • Always re-enable the clusters that rolling maintenance disabled when it aborts.
  • Fail fast instead of waiting for the full timeout when all migrations have failed.
主要语言
Java
星标
3.1k
派生
1.4k
平均合并
7 天 16 小时
30 天内合并 PR
15

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/cloudstack 的其他 Issue

查看 apache/cloudstack 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。