KVM: live migration fails with NullPointerException in `LibvirtMigrateCommandWrapper.updateVmSharesIfNeeded` when the running domain XML has no `<shares>` element

Offen
  • Stimmt etwas nicht?
#14,652 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@HawkForge arbeitet bereits daran.

Seit 12.11.2026.

  • #14685 von @HawkForge — offen

Bewertung

Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Anfängerfreundlichkeit
68/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
java
Bereich
backend, infrastructure

Rechercherichtung

Read LibvirtMigrateCommandWrapper.updateVmSharesIfNeeded and trace how newVmCpuShares is passed from PrepareForMigrationAnswer; inspect the existing XML handling and nearby tests. Reproduce with a domain XML that has no <shares> element, then verify migration preparation handles that case and preserves the requested destination shares without a NullPointerException.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

bug
problem

Live migration from a cgroup v1 host to a cgroup v2 host fails with a NullPointerException on the source agent, before libvirt is called, when the running domain XML has no <cputune><shares> element.

updateVmSharesIfNeeded assumes <shares> is always present:

Node sharesNode = root.getElementsByTagName("shares").item(0);
String currentShares = sharesNode.getTextContent();   // NPE when <shares> is missing

However, CloudStack itself does not always write <shares>. CpuTuneDef.toString() emits it only when shares > 0, and calculateCpuShares() returns vCPUs * minSpeed, where minSpeed = (int)(speed / cpuOvercommitRatio). When speed < cpuOvercommitRatio (e.g. 100 MHz offering, overcommit 1000), minSpeed is truncated to 0, so the VM starts with no <shares> element and libvirt leaves the cgroup default (1024 on v1, 100 on v2).

The NPE then triggers when, at migration time, the shares computed by the source agent (calculateCpuShares(vmTO)) differ from newVmCpuShares returned by the destination. This happens, for example, when:

  • the cluster overcommit ratio has been lowered since the VM was started (source now computes vCPUs * minSpeed > 0), and/or
  • the destination host uses a different cgroup version (v2 rescaling yields a different value).

Between two cgroup v1 hosts the values usually match, so the method returns early without parsing the XML and the bug stays hidden. It surfaces when operators start adding cgroup v2 hosts (EL9/EL10, Ubuntu 22.04+) to existing clusters.

In our case 58 of 88 VMs on a single EL8 host had no <shares> (all started while the cluster overcommit was 1000), and none of them could be live-migrated to EL9 hosts.

versions

Observed on 4.19.1.3.
Same code (no null check on the node in updateVmSharesIfNeeded) verified in:
4.19.2.0, 4.19.3.0, 4.20.0.0 - 4.20.3.1, 4.21.0.0, 4.22.0.0 - 4.22.1.1, 4.23.0.0,
and the current 4.19, 4.20, 4.22 and main branches.

CONFIGURATION

Advanced zone, KVM cluster with mixed hosts during an OS upgrade:

  • source hosts: Oracle Linux 8 (cgroup v1), libvirt 8.0.0, qemu-kvm 6.2.0
  • destination hosts: Oracle Linux 9 (cgroup v2)

Cluster cpuOvercommitRatio was previously 1000, later lowered to 10. Compute offerings use 100 MHz (system router offering: 500 MHz).

The steps to reproduce the bug
  1. KVM cluster with a cgroup v1 host (e.g. EL8). Set the cluster cpuOvercommitRatio
    higher than the offering CPU speed (e.g. ratio 1000, offering 1 vCPU @ 100 MHz).
  2. Start a VM on the cgroup v1 host.
    -> virsh dumpxml <vm> shows no inside (or no at all).
  3. Lower the cluster cpuOvercommitRatio (e.g. to 10).
  4. Add a cgroup v2 host (e.g. EL9) to the same cluster.
  5. Live migrate the VM from the v1 host to the v2 host.
EXPECTED RESULTS

The VM is migrated. If is missing from the domain XML, the agent adds it
with the value computed for the destination (newVmCpuShares).

ACTUAL RESULTS

Management server:

PrepareForMigrationAnswer ... "newVmCpuShares":"1"
Setting CPU shares to [1] as part of migrate command for VM [VM {... name: "i-X-Y-VM" ...}]
...
Migration was unsuccessful.
Unable to migrate vm i-X-Y-VM from host <source-host>

Source host agent:

WARN  [cloud.agent.Agent] (agentRequest-Handler-2:null) Caught:
java.lang.NullPointerException
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.updateVmSharesIfNeeded(LibvirtMigrateCommandWrapper.java:436)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:229)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:89)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)
        at com.cloud.agent.Agent.processRequest(Agent.java:683)
        ...

Domain state on the source host:

# virsh dumpxml i-X-Y-VM | grep -c '<shares>'
0
# virsh schedinfo i-X-Y-VM | grep cpu_shares
cpu_shares     : 1024
What to do about it?
SUGGESTED FIX

In updateVmSharesIfNeeded:

  • if newVmCpuShares is null or <= 0, return the XML unchanged;
  • if <shares> is missing, create it (and <cputune> if needed) instead of dereferencing null.

Rough sketch:

Integer newVmCpuShares = migrateCommand.getNewVmCpuShares();
if (newVmCpuShares == null || newVmCpuShares <= 0) {
    return xmlDesc;
}
int currentCpuShares = libvirtComputingResource.calculateCpuShares(migrateCommand.getVirtualMachine());
if (newVmCpuShares == currentCpuShares) {
    return xmlDesc;
}
...
Element root = document.getDocumentElement();
Node sharesNode = root.getElementsByTagName("shares").item(0);
if (sharesNode == null) {
    Node cputune = root.getElementsByTagName("cputune").item(0);
    if (cputune == null) {
        cputune = document.createElement("cputune");
        root.appendChild(cputune);
    }
    sharesNode = document.createElement("shares");
    cputune.appendChild(sharesNode);
}
String currentShares = sharesNode.getTextContent();
...

The null check on newVmCpuShares also avoids an unboxing NPE in newVmCpuShares == currentCpuShares, e.g. when the destination agent is older and does not populate newVmCpuShares in PrepareForMigrationAnswer.

A related improvement could be to never compute minSpeed/shares as 0 (e.g. max(1, ...)), so that <shares> is always present and VMs keep a proportional weight with high overcommit ratios. Today all such VMs fall back to the cgroup default, regardless of size.

WORKAROUND

Before migrating, add <shares> to the live definition on the source host, e.g.:

virsh schedinfo <vm> --live --set cpu_shares=<value>

Any positive value works, since the agent then replaces it with newVmCpuShares during migration.

Vorherrschende Sprache
Java
Sterne
3.1k
Forks
1.4k
Ø Merge
7 T. 16 Std.
Gemergte PRs (30 T.)
15

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus apache/cloudstack

Alle Issues in apache/cloudstack

Ähnliche Issues

Weitere Issues zu Java

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.