Interrupted running-VM volume snapshot leaves the VM on an orphan overlay, disk reverts on next start

Abierto
  • ¿Algo no está bien?
#14,691 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
15/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
java
Área
backend, infrastructure

Línea de trabajo

Start in KVMStorageProcessor, following the volume snapshot path to mergeSnapshotIntoBaseFileWithoutEvents(), and check how Script.runScript() turns a non-zero virsh blockcommit exit into a null result. Agent death is handled in Agent.java at the ReadyCommand System.exit(1). Done means a failed merge fails the snapshot rather than marking it BackedUp, and the design for recovering from an orphan overlay is agreed in the thread first.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
problem

A volume snapshot (non-incremental) of a running VM does three steps in KVMStorageProcessor:

  1. snapshotCreateXML(DISK_ONLY) moves the disk onto a new overlay /mnt/<pool>/<random uuid> (instant).
  2. qemu-img convert copies the frozen base to the snapshot location (minutes to hours, depends on volume size).
  3. virsh blockcommit --active --wait --pivot merges the overlay back into the base.

If step 3 never starts, the VM keeps running on the overlay while volumes.path still points to the base.
The next stop/start builds the domain from volumes.path, so every write since the snapshot started is silently
lost (it only exists in the orphan overlay). After a start the base is written again, so the overlay can no longer
simply be committed back.

There are two ways step 3 never starts.

A. The agent JVM goes away during the convert.

  • SIGKILL / OOM kill of the agent.
  • The agent's own System.exit(1) in Agent.java (ReadyCommand with details) after a failed reconnect. Reproduced by
    making the management server fail the reconnect with Unable to acquire lock on host while the convert was running.

B. virsh blockcommit fails before the job starts, and the failure is reported as success.
Script.runScript() returns null when execute() returns an error, so mergeSnapshotIntoBaseFileWithoutEvents()
gets null for any non-zero exit and logs Successfully merged snapshot. The snapshot ends BackedUp.
Reproduced with:

  • a NAS backup (backup-begin) started during the convert and still running at merge time:
    Operation not supported: disk 'vdb' already in active block job;
  • the domain job held by another operation for more than 30 s:
    Timed out during operation: cannot acquire state change lock.

NAS backups do not go through the VM work queue and do not check for volumes in Snapshotting, so nothing prevents
this overlap.

versions

ACS 4.20+

The steps to reproduce the bug
  1. Running VM with a data disk on NFS, write some data so the convert takes a while.
  2. Take a volume snapshot of the data disk.
  3. As soon as virsh domblklist shows the disk on a new bare-UUID file, either:
    • kill -9 the agent JVM; or
    • start a backup: virsh backup-begin <vm> (push or pull mode) and keep it running until the merge.
  4. Write data in the guest, then stop and start the VM from CloudStack.
What to do about it?

Expected:
The merge either completes, or the snapshot fails and the VM is not left on an overlay that CloudStack does not track.

Actual:
The VM runs on /mnt/<pool>/<uuid> (backing file = the volume). Case B logs:

WARN  Process [N] for command [/bin/bash -c virsh blockcommit <vm> vdb --base ... --pivot ] encountered the error: [error: Operation not supported: disk 'vdb' already in active block job].
DEBUG Successfully merged snapshot [<overlay uuid>] into VM [<vm>] ...

Later snapshots of the volume fail with VM [...] has no disk with path [<base>].
After stop/start the guest reads the data from the time the snapshot started.

Proposed fix

  1. Detect merge failures. Make Script.runScript() (or at least the merge call) return an error on a non-zero
    exit. After the merge, check that the domain disk source is the base path again. If the disk has another block
    job, wait and retry the commit. If it still fails, fail the snapshot with a clear error instead of BackedUp.
  2. Recover after agent death. Before creating the overlay, write a marker (vm, disk, base, overlay) on the shared primary pool, keyed by volume. On every VM start, and on agent start once the pools are mounted, any host that finds a marker reconciles the volume:
  • Domain still running on the overlay: run blockcommit --active --pivot, or pivot the commit job that is already running.
  • Domain gone: commit the overlay offline with qemu-img commit, then start.
  • Overlay still locked by a live qemu: refuse to start.
  • Base written since the marker: never commit, raise an alert and keep the overlay.
    Remove the marker only after domblklist shows the base path again.
  1. Do not exit with commands in progress. On a ReadyCommand with details, reconnect (as isExitOnFailures()=false
    already does) instead of System.exit(1), at least while commands are in progress.
  2. Serialize NAS backups with volume snapshots of the same VM (VM work queue or refuse while a volume is
    Snapshotting).
Lenguaje dominante
Java
Estrellas
3.1k
Forks
1.4k
Merge medio
7 d 16 h
PR fusionados (30 d)
15

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/cloudstack

Todos los issues de apache/cloudstack

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.