Interrupted running-VM volume snapshot leaves the VM on an orphan overlay, disk reverts on next start

Start in KVMStorageProcessor, following the volume snapshot path to mergeSnapshotIntoBaseFileWithoutEvents(), and check how Script.runScript() turns a non-zero virsh blockcommit exit into a null result. Agent death is handled in Agent.java at the ReadyCommand System.exit(1). Done means a failed merge fails the snapshot rather than marking it BackedUp, and the design for recovering from an orphan overlay is agreed in the thread first.

problem

A volume snapshot (non-incremental) of a running VM does three steps in KVMStorageProcessor:

1. snapshotCreateXML(DISK_ONLY) moves the disk onto a new overlay /mnt/<pool>/<random uuid> (instant).
2. qemu-img convert copies the frozen base to the snapshot location (minutes to hours, depends on volume size).
3. virsh blockcommit --active --wait --pivot merges the overlay back into the base.

If step 3 never starts, the VM keeps running on the overlay while volumes.path still points to the base.
The next stop/start builds the domain from volumes.path, so every write since the snapshot started is silently
lost (it only exists in the orphan overlay). After a start the base is written again, so the overlay can no longer
simply be committed back.

There are two ways step 3 never starts.

A. The agent JVM goes away during the convert.

• SIGKILL / OOM kill of the agent.
• The agent's own System.exit(1) in Agent.java (ReadyCommand with details) after a failed reconnect. Reproduced by
  making the management server fail the reconnect with Unable to acquire lock on host while the convert was running.

B. virsh blockcommit fails before the job starts, and the failure is reported as success.
Script.runScript() returns null when execute() returns an error, so mergeSnapshotIntoBaseFileWithoutEvents()
gets null for any non-zero exit and logs Successfully merged snapshot. The snapshot ends BackedUp.
Reproduced with:

• a NAS backup (backup-begin) started during the convert and still running at merge time:
  Operation not supported: disk 'vdb' already in active block job;
• the domain job held by another operation for more than 30 s:
  Timed out during operation: cannot acquire state change lock.

NAS backups do not go through the VM work queue and do not check for volumes in Snapshotting, so nothing prevents
this overlap.

versions

ACS 4.20+

The steps to reproduce the bug

1. Running VM with a data disk on NFS, write some data so the convert takes a while.
2. Take a volume snapshot of the data disk.
3. As soon as virsh domblklist shows the disk on a new bare-UUID file, either:
   • kill -9 the agent JVM; or
   • start a backup: virsh backup-begin <vm> (push or pull mode) and keep it running until the merge.
4. Write data in the guest, then stop and start the VM from CloudStack.

What to do about it?

Expected:
The merge either completes, or the snapshot fails and the VM is not left on an overlay that CloudStack does not track.

Actual:
The VM runs on /mnt/<pool>/<uuid> (backing file = the volume). Case B logs:

WARN  Process [N] for command [/bin/bash -c virsh blockcommit <vm> vdb --base ... --pivot ] encountered the error: [error: Operation not supported: disk 'vdb' already in active block job].
DEBUG Successfully merged snapshot [<overlay uuid>] into VM [<vm>] ...

Later snapshots of the volume fail with VM [...] has no disk with path [<base>].
After stop/start the guest reads the data from the time the snapshot started.

Proposed fix

1. Detect merge failures. Make Script.runScript() (or at least the merge call) return an error on a non-zero
   exit. After the merge, check that the domain disk source is the base path again. If the disk has another block
   job, wait and retry the commit. If it still fails, fail the snapshot with a clear error instead of BackedUp.
2. Recover after agent death. Before creating the overlay, write a marker (vm, disk, base, overlay) on the shared primary pool, keyed by volume. On every VM start, and on agent start once the pools are mounted, any host that finds a marker reconciles the volume:

• Domain still running on the overlay: run blockcommit --active --pivot, or pivot the commit job that is already running.
• Domain gone: commit the overlay offline with qemu-img commit, then start.
• Overlay still locked by a live qemu: refuse to start.
• Base written since the marker: never commit, raise an alert and keep the overlay.
  Remove the marker only after domblklist shows the base path again.

3. Do not exit with commands in progress. On a ReadyCommand with details, reconnect (as isExitOnFailures()=false
   already does) instead of System.exit(1), at least while commands are in progress.
4. Serialize NAS backups with volume snapshots of the same VM (VM work queue or refuse while a volume is
   Snapshotting).