Interrupted running-VM volume snapshot leaves the VM on an orphan overlay, disk reverts on next start

Aberta
  • Algo errado?
#14,691 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
15/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
java
Domínio
backend, infrastructure

Direção de pesquisa

Start in KVMStorageProcessor, following the volume snapshot path to mergeSnapshotIntoBaseFileWithoutEvents(), and check how Script.runScript() turns a non-zero virsh blockcommit exit into a null result. Agent death is handled in Agent.java at the ReadyCommand System.exit(1). Done means a failed merge fails the snapshot rather than marking it BackedUp, and the design for recovering from an orphan overlay is agreed in the thread first.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

bug
problem

A volume snapshot (non-incremental) of a running VM does three steps in KVMStorageProcessor:

  1. snapshotCreateXML(DISK_ONLY) moves the disk onto a new overlay /mnt/<pool>/<random uuid> (instant).
  2. qemu-img convert copies the frozen base to the snapshot location (minutes to hours, depends on volume size).
  3. virsh blockcommit --active --wait --pivot merges the overlay back into the base.

If step 3 never starts, the VM keeps running on the overlay while volumes.path still points to the base.
The next stop/start builds the domain from volumes.path, so every write since the snapshot started is silently
lost (it only exists in the orphan overlay). After a start the base is written again, so the overlay can no longer
simply be committed back.

There are two ways step 3 never starts.

A. The agent JVM goes away during the convert.

  • SIGKILL / OOM kill of the agent.
  • The agent's own System.exit(1) in Agent.java (ReadyCommand with details) after a failed reconnect. Reproduced by
    making the management server fail the reconnect with Unable to acquire lock on host while the convert was running.

B. virsh blockcommit fails before the job starts, and the failure is reported as success.
Script.runScript() returns null when execute() returns an error, so mergeSnapshotIntoBaseFileWithoutEvents()
gets null for any non-zero exit and logs Successfully merged snapshot. The snapshot ends BackedUp.
Reproduced with:

  • a NAS backup (backup-begin) started during the convert and still running at merge time:
    Operation not supported: disk 'vdb' already in active block job;
  • the domain job held by another operation for more than 30 s:
    Timed out during operation: cannot acquire state change lock.

NAS backups do not go through the VM work queue and do not check for volumes in Snapshotting, so nothing prevents
this overlap.

versions

ACS 4.20+

The steps to reproduce the bug
  1. Running VM with a data disk on NFS, write some data so the convert takes a while.
  2. Take a volume snapshot of the data disk.
  3. As soon as virsh domblklist shows the disk on a new bare-UUID file, either:
    • kill -9 the agent JVM; or
    • start a backup: virsh backup-begin <vm> (push or pull mode) and keep it running until the merge.
  4. Write data in the guest, then stop and start the VM from CloudStack.
What to do about it?

Expected:
The merge either completes, or the snapshot fails and the VM is not left on an overlay that CloudStack does not track.

Actual:
The VM runs on /mnt/<pool>/<uuid> (backing file = the volume). Case B logs:

WARN  Process [N] for command [/bin/bash -c virsh blockcommit <vm> vdb --base ... --pivot ] encountered the error: [error: Operation not supported: disk 'vdb' already in active block job].
DEBUG Successfully merged snapshot [<overlay uuid>] into VM [<vm>] ...

Later snapshots of the volume fail with VM [...] has no disk with path [<base>].
After stop/start the guest reads the data from the time the snapshot started.

Proposed fix

  1. Detect merge failures. Make Script.runScript() (or at least the merge call) return an error on a non-zero
    exit. After the merge, check that the domain disk source is the base path again. If the disk has another block
    job, wait and retry the commit. If it still fails, fail the snapshot with a clear error instead of BackedUp.
  2. Recover after agent death. Before creating the overlay, write a marker (vm, disk, base, overlay) on the shared primary pool, keyed by volume. On every VM start, and on agent start once the pools are mounted, any host that finds a marker reconciles the volume:
  • Domain still running on the overlay: run blockcommit --active --pivot, or pivot the commit job that is already running.
  • Domain gone: commit the overlay offline with qemu-img commit, then start.
  • Overlay still locked by a live qemu: refuse to start.
  • Base written since the marker: never commit, raise an alert and keep the overlay.
    Remove the marker only after domblklist shows the base path again.
  1. Do not exit with commands in progress. On a ReadyCommand with details, reconnect (as isExitOnFailures()=false
    already does) instead of System.exit(1), at least while commands are in progress.
  2. Serialize NAS backups with volume snapshots of the same VM (VM work queue or refuse while a volume is
    Snapshotting).
Linguagem predominante
Java
Estrelas
3.1k
Forks
1.4k
Merge médio
7d 16h
PRs com merge (30d)
15

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/cloudstack

Todas as issues de apache/cloudstack

Issues semelhantes

Mais issues de Java

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.