KVM: live migration fails with NullPointerException in `LibvirtMigrateCommandWrapper.updateVmSharesIfNeeded` when the running domain XML has no `<shares>` element

Aberta
  • Algo errado?
#14,652 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

@HawkForge já está trabalhando nisso.

Desde 12/11/2026.

  • #14685 de @HawkForge — aberto

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
68/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
java
Domínio
backend, infrastructure

Direção de pesquisa

Read LibvirtMigrateCommandWrapper.updateVmSharesIfNeeded and trace how newVmCpuShares is passed from PrepareForMigrationAnswer; inspect the existing XML handling and nearby tests. Reproduce with a domain XML that has no <shares> element, then verify migration preparation handles that case and preserves the requested destination shares without a NullPointerException.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

bug
problem

Live migration from a cgroup v1 host to a cgroup v2 host fails with a NullPointerException on the source agent, before libvirt is called, when the running domain XML has no <cputune><shares> element.

updateVmSharesIfNeeded assumes <shares> is always present:

Node sharesNode = root.getElementsByTagName("shares").item(0);
String currentShares = sharesNode.getTextContent();   // NPE when <shares> is missing

However, CloudStack itself does not always write <shares>. CpuTuneDef.toString() emits it only when shares > 0, and calculateCpuShares() returns vCPUs * minSpeed, where minSpeed = (int)(speed / cpuOvercommitRatio). When speed < cpuOvercommitRatio (e.g. 100 MHz offering, overcommit 1000), minSpeed is truncated to 0, so the VM starts with no <shares> element and libvirt leaves the cgroup default (1024 on v1, 100 on v2).

The NPE then triggers when, at migration time, the shares computed by the source agent (calculateCpuShares(vmTO)) differ from newVmCpuShares returned by the destination. This happens, for example, when:

  • the cluster overcommit ratio has been lowered since the VM was started (source now computes vCPUs * minSpeed > 0), and/or
  • the destination host uses a different cgroup version (v2 rescaling yields a different value).

Between two cgroup v1 hosts the values usually match, so the method returns early without parsing the XML and the bug stays hidden. It surfaces when operators start adding cgroup v2 hosts (EL9/EL10, Ubuntu 22.04+) to existing clusters.

In our case 58 of 88 VMs on a single EL8 host had no <shares> (all started while the cluster overcommit was 1000), and none of them could be live-migrated to EL9 hosts.

versions

Observed on 4.19.1.3.
Same code (no null check on the node in updateVmSharesIfNeeded) verified in:
4.19.2.0, 4.19.3.0, 4.20.0.0 - 4.20.3.1, 4.21.0.0, 4.22.0.0 - 4.22.1.1, 4.23.0.0,
and the current 4.19, 4.20, 4.22 and main branches.

CONFIGURATION

Advanced zone, KVM cluster with mixed hosts during an OS upgrade:

  • source hosts: Oracle Linux 8 (cgroup v1), libvirt 8.0.0, qemu-kvm 6.2.0
  • destination hosts: Oracle Linux 9 (cgroup v2)

Cluster cpuOvercommitRatio was previously 1000, later lowered to 10. Compute offerings use 100 MHz (system router offering: 500 MHz).

The steps to reproduce the bug
  1. KVM cluster with a cgroup v1 host (e.g. EL8). Set the cluster cpuOvercommitRatio
    higher than the offering CPU speed (e.g. ratio 1000, offering 1 vCPU @ 100 MHz).
  2. Start a VM on the cgroup v1 host.
    -> virsh dumpxml <vm> shows no inside (or no at all).
  3. Lower the cluster cpuOvercommitRatio (e.g. to 10).
  4. Add a cgroup v2 host (e.g. EL9) to the same cluster.
  5. Live migrate the VM from the v1 host to the v2 host.
EXPECTED RESULTS

The VM is migrated. If is missing from the domain XML, the agent adds it
with the value computed for the destination (newVmCpuShares).

ACTUAL RESULTS

Management server:

PrepareForMigrationAnswer ... "newVmCpuShares":"1"
Setting CPU shares to [1] as part of migrate command for VM [VM {... name: "i-X-Y-VM" ...}]
...
Migration was unsuccessful.
Unable to migrate vm i-X-Y-VM from host <source-host>

Source host agent:

WARN  [cloud.agent.Agent] (agentRequest-Handler-2:null) Caught:
java.lang.NullPointerException
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.updateVmSharesIfNeeded(LibvirtMigrateCommandWrapper.java:436)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:229)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:89)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)
        at com.cloud.agent.Agent.processRequest(Agent.java:683)
        ...

Domain state on the source host:

# virsh dumpxml i-X-Y-VM | grep -c '<shares>'
0
# virsh schedinfo i-X-Y-VM | grep cpu_shares
cpu_shares     : 1024
What to do about it?
SUGGESTED FIX

In updateVmSharesIfNeeded:

  • if newVmCpuShares is null or <= 0, return the XML unchanged;
  • if <shares> is missing, create it (and <cputune> if needed) instead of dereferencing null.

Rough sketch:

Integer newVmCpuShares = migrateCommand.getNewVmCpuShares();
if (newVmCpuShares == null || newVmCpuShares <= 0) {
    return xmlDesc;
}
int currentCpuShares = libvirtComputingResource.calculateCpuShares(migrateCommand.getVirtualMachine());
if (newVmCpuShares == currentCpuShares) {
    return xmlDesc;
}
...
Element root = document.getDocumentElement();
Node sharesNode = root.getElementsByTagName("shares").item(0);
if (sharesNode == null) {
    Node cputune = root.getElementsByTagName("cputune").item(0);
    if (cputune == null) {
        cputune = document.createElement("cputune");
        root.appendChild(cputune);
    }
    sharesNode = document.createElement("shares");
    cputune.appendChild(sharesNode);
}
String currentShares = sharesNode.getTextContent();
...

The null check on newVmCpuShares also avoids an unboxing NPE in newVmCpuShares == currentCpuShares, e.g. when the destination agent is older and does not populate newVmCpuShares in PrepareForMigrationAnswer.

A related improvement could be to never compute minSpeed/shares as 0 (e.g. max(1, ...)), so that <shares> is always present and VMs keep a proportional weight with high overcommit ratios. Today all such VMs fall back to the cgroup default, regardless of size.

WORKAROUND

Before migrating, add <shares> to the live definition on the source host, e.g.:

virsh schedinfo <vm> --live --set cpu_shares=<value>

Any positive value works, since the agent then replaces it with newVmCpuShares during migration.

Linguagem predominante
Java
Estrelas
3.1k
Forks
1.4k
Merge médio
7d 16h
PRs com merge (30d)
15

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/cloudstack

Todas as issues de apache/cloudstack

Issues semelhantes

Mais issues de Java

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.