KVM: live migration fails with NullPointerException in `LibvirtMigrateCommandWrapper.updateVmSharesIfNeeded` when the running domain XML has no `<shares>` element

Aperta
  • Qualcosa non va?
#14,652 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@HawkForge ci sta già lavorando.

Dal 12/11/2026.

  • #14685 di @HawkForge — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
68/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
java
Ambito
backend, infrastructure

Direzione di ricerca

Read LibvirtMigrateCommandWrapper.updateVmSharesIfNeeded and trace how newVmCpuShares is passed from PrepareForMigrationAnswer; inspect the existing XML handling and nearby tests. Reproduce with a domain XML that has no <shares> element, then verify migration preparation handles that case and preserves the requested destination shares without a NullPointerException.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
problem

Live migration from a cgroup v1 host to a cgroup v2 host fails with a NullPointerException on the source agent, before libvirt is called, when the running domain XML has no <cputune><shares> element.

updateVmSharesIfNeeded assumes <shares> is always present:

Node sharesNode = root.getElementsByTagName("shares").item(0);
String currentShares = sharesNode.getTextContent();   // NPE when <shares> is missing

However, CloudStack itself does not always write <shares>. CpuTuneDef.toString() emits it only when shares > 0, and calculateCpuShares() returns vCPUs * minSpeed, where minSpeed = (int)(speed / cpuOvercommitRatio). When speed < cpuOvercommitRatio (e.g. 100 MHz offering, overcommit 1000), minSpeed is truncated to 0, so the VM starts with no <shares> element and libvirt leaves the cgroup default (1024 on v1, 100 on v2).

The NPE then triggers when, at migration time, the shares computed by the source agent (calculateCpuShares(vmTO)) differ from newVmCpuShares returned by the destination. This happens, for example, when:

  • the cluster overcommit ratio has been lowered since the VM was started (source now computes vCPUs * minSpeed > 0), and/or
  • the destination host uses a different cgroup version (v2 rescaling yields a different value).

Between two cgroup v1 hosts the values usually match, so the method returns early without parsing the XML and the bug stays hidden. It surfaces when operators start adding cgroup v2 hosts (EL9/EL10, Ubuntu 22.04+) to existing clusters.

In our case 58 of 88 VMs on a single EL8 host had no <shares> (all started while the cluster overcommit was 1000), and none of them could be live-migrated to EL9 hosts.

versions

Observed on 4.19.1.3.
Same code (no null check on the node in updateVmSharesIfNeeded) verified in:
4.19.2.0, 4.19.3.0, 4.20.0.0 - 4.20.3.1, 4.21.0.0, 4.22.0.0 - 4.22.1.1, 4.23.0.0,
and the current 4.19, 4.20, 4.22 and main branches.

CONFIGURATION

Advanced zone, KVM cluster with mixed hosts during an OS upgrade:

  • source hosts: Oracle Linux 8 (cgroup v1), libvirt 8.0.0, qemu-kvm 6.2.0
  • destination hosts: Oracle Linux 9 (cgroup v2)

Cluster cpuOvercommitRatio was previously 1000, later lowered to 10. Compute offerings use 100 MHz (system router offering: 500 MHz).

The steps to reproduce the bug
  1. KVM cluster with a cgroup v1 host (e.g. EL8). Set the cluster cpuOvercommitRatio
    higher than the offering CPU speed (e.g. ratio 1000, offering 1 vCPU @ 100 MHz).
  2. Start a VM on the cgroup v1 host.
    -> virsh dumpxml <vm> shows no inside (or no at all).
  3. Lower the cluster cpuOvercommitRatio (e.g. to 10).
  4. Add a cgroup v2 host (e.g. EL9) to the same cluster.
  5. Live migrate the VM from the v1 host to the v2 host.
EXPECTED RESULTS

The VM is migrated. If is missing from the domain XML, the agent adds it
with the value computed for the destination (newVmCpuShares).

ACTUAL RESULTS

Management server:

PrepareForMigrationAnswer ... "newVmCpuShares":"1"
Setting CPU shares to [1] as part of migrate command for VM [VM {... name: "i-X-Y-VM" ...}]
...
Migration was unsuccessful.
Unable to migrate vm i-X-Y-VM from host <source-host>

Source host agent:

WARN  [cloud.agent.Agent] (agentRequest-Handler-2:null) Caught:
java.lang.NullPointerException
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.updateVmSharesIfNeeded(LibvirtMigrateCommandWrapper.java:436)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:229)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:89)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)
        at com.cloud.agent.Agent.processRequest(Agent.java:683)
        ...

Domain state on the source host:

# virsh dumpxml i-X-Y-VM | grep -c '<shares>'
0
# virsh schedinfo i-X-Y-VM | grep cpu_shares
cpu_shares     : 1024
What to do about it?
SUGGESTED FIX

In updateVmSharesIfNeeded:

  • if newVmCpuShares is null or <= 0, return the XML unchanged;
  • if <shares> is missing, create it (and <cputune> if needed) instead of dereferencing null.

Rough sketch:

Integer newVmCpuShares = migrateCommand.getNewVmCpuShares();
if (newVmCpuShares == null || newVmCpuShares <= 0) {
    return xmlDesc;
}
int currentCpuShares = libvirtComputingResource.calculateCpuShares(migrateCommand.getVirtualMachine());
if (newVmCpuShares == currentCpuShares) {
    return xmlDesc;
}
...
Element root = document.getDocumentElement();
Node sharesNode = root.getElementsByTagName("shares").item(0);
if (sharesNode == null) {
    Node cputune = root.getElementsByTagName("cputune").item(0);
    if (cputune == null) {
        cputune = document.createElement("cputune");
        root.appendChild(cputune);
    }
    sharesNode = document.createElement("shares");
    cputune.appendChild(sharesNode);
}
String currentShares = sharesNode.getTextContent();
...

The null check on newVmCpuShares also avoids an unboxing NPE in newVmCpuShares == currentCpuShares, e.g. when the destination agent is older and does not populate newVmCpuShares in PrepareForMigrationAnswer.

A related improvement could be to never compute minSpeed/shares as 0 (e.g. max(1, ...)), so that <shares> is always present and VMs keep a proportional weight with high overcommit ratios. Today all such VMs fall back to the cgroup default, regardless of size.

WORKAROUND

Before migrating, add <shares> to the live definition on the source host, e.g.:

virsh schedinfo <vm> --live --set cpu_shares=<value>

Any positive value works, since the agent then replaces it with newVmCpuShares during migration.

Lingua principale
Java
Stelle
3.1k
Fork
1.4k
Merge medio
7g 16h
PR unite (30g)
15

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/cloudstack

Tutte le issue di apache/cloudstack

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.