Rolling maintenance fails with ErrorInMaintenance because the cluster it disables is excluded as VM migration destination

Aberta
  • Algo errado?
#14,516 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
42/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
java
Domínio
backend, infrastructure

Direção de pesquisa

Start with RollingMaintenanceManagerImpl.disableClusterIfEnabled() and startRollingMaintenanceHostInCluster(), then trace how DeploymentPlanningManagerImpl filters disabled clusters for VmWorkMigrateAway. Reproduce in a two-host, single-cluster zone with migrate.vm.across.clusters=false and inspect migration tests around rolling maintenance and destination selection. Done means VMs can evacuate to the remaining host, maintenance completes, and an aborted run does not leave the cluster disabled.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

bug
problem

startRollingMaintenance disables the cluster itself (RollingMaintenanceManagerImpl.disableClusterIfEnabled() sets the allocation state to Disabled) before putting hosts into maintenance. The VMs on the host then cannot be migrated to the other host in the same cluster, because the deployment planner excludes Disabled clusters. The host ends up in ErrorInMaintenance after the maintenance timeout, and the cluster is left in Disabled state.

Management server log:

DEBUG [c.c.d.DeploymentPlanningManagerImpl] Adding clusters [1] of pod [1] to the void set because these clusters are in the Disabled state.
WARN  [c.c.d.DeploymentPlanningManagerImpl] VM last host ID: 2 belongs to zone ID: 1 for which config - migrate.vm.across.clusters is false and storage migration would be needed for inter-cluster migration, therefore, adding all other clusters except ID: 1 from this zone to avoid list
WARN  [c.c.v.ClusteredVirtualMachineManagerImpl] Unable to find destination for migrating the vm VM instance {"id":786,"instanceName":"i-2-786-VM","type":"User",...}
ERROR [c.c.v.VmWorkJobDispatcher] Unable to complete AsyncJob {"cmd":"com.cloud.vm.VmWorkMigrateAway",...}
      InsufficientServerCapacityException: Unable to find a server to migrate to.Scope=interface com.cloud.dc.DataCenter; id=1
...
ERROR [c.c.r.RollingMaintenanceManagerImpl] Timeout: waited 1800000ms for host <uuid>(<host>) to be in Maintenance state, but after timeout it is in ErrorInMaintenance
    at RollingMaintenanceManagerImpl.waitForHostInMaintenance(RollingMaintenanceManagerImpl.java:775)
    at RollingMaintenanceManagerImpl.putHostIntoMaintenance(RollingMaintenanceManagerImpl.java:411)
    at RollingMaintenanceManagerImpl.startRollingMaintenanceHostInCluster(RollingMaintenanceManagerImpl.java:327)
    at RollingMaintenanceManagerImpl.startRollingMaintenance(RollingMaintenanceManagerImpl.java:210)

The same migrate-away failure occurs for the system VMs, the virtual router and the user VMs on the host.

Root cause (from code and logs):

  • startRollingMaintenance calls disableClusterIfEnabled() before startRollingMaintenanceHostInCluster().
  • DeploymentPlanningManagerImpl voids Disabled clusters, so with migrate.vm.across.clusters=false (or a single-cluster zone) no destination is left.
  • attemptMaintain() then finds running VMs and no pending migration retries, and calls setHostIntoErrorInMaintenance().
versions
  • CloudStack: main branch (4.22.1.0-SNAPSHOT)
  • Hypervisor: KVM (Ubuntu 26.04)
  • Zone with a single cluster of 2 hosts, migrate.vm.across.clusters=false
The steps to reproduce the bug
  1. Have an Enabled cluster with 2 KVM hosts, with system VMs, a VR and user VMs running on the second host.
  2. Run startRollingMaintenance on the cluster (cmk: start rollingmaintenance clusterids=<cluster id>).
  3. Wait for the first host to enter maintenance.
What to do about it?

Expected: VMs are migrated to the other host, the host reaches Maintenance, rolling maintenance continues, and the cluster is re-enabled at the end.

Actual: the host goes to ErrorInMaintenance after the timeout and the cluster stays Disabled.

Possible fixes:

  • Allow migrations triggered by host maintenance (VmWorkMigrateAway) to target hosts in a cluster disabled by rolling maintenance, or do not use the Disabled allocation state to block new deployments.
  • Always re-enable the clusters that rolling maintenance disabled when it aborts.
  • Fail fast instead of waiting for the full timeout when all migrations have failed.
Linguagem predominante
Java
Estrelas
3.1k
Forks
1.4k
Merge médio
7d 16h
PRs com merge (30d)
15

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/cloudstack

Todas as issues de apache/cloudstack

Issues semelhantes

Mais issues de Java

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.