Interrupted running-VM volume snapshot leaves the VM on an orphan overlay, disk reverts on next start

Đang mở
  • Có gì không ổn?
#14,691 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
15/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
java
Lĩnh vực
backend, infrastructure

Hướng nghiên cứu

Start in KVMStorageProcessor, following the volume snapshot path to mergeSnapshotIntoBaseFileWithoutEvents(), and check how Script.runScript() turns a non-zero virsh blockcommit exit into a null result. Agent death is handled in Agent.java at the ReadyCommand System.exit(1). Done means a failed merge fails the snapshot rather than marking it BackedUp, and the design for recovering from an orphan overlay is agreed in the thread first.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug
problem

A volume snapshot (non-incremental) of a running VM does three steps in KVMStorageProcessor:

  1. snapshotCreateXML(DISK_ONLY) moves the disk onto a new overlay /mnt/<pool>/<random uuid> (instant).
  2. qemu-img convert copies the frozen base to the snapshot location (minutes to hours, depends on volume size).
  3. virsh blockcommit --active --wait --pivot merges the overlay back into the base.

If step 3 never starts, the VM keeps running on the overlay while volumes.path still points to the base.
The next stop/start builds the domain from volumes.path, so every write since the snapshot started is silently
lost (it only exists in the orphan overlay). After a start the base is written again, so the overlay can no longer
simply be committed back.

There are two ways step 3 never starts.

A. The agent JVM goes away during the convert.

  • SIGKILL / OOM kill of the agent.
  • The agent's own System.exit(1) in Agent.java (ReadyCommand with details) after a failed reconnect. Reproduced by
    making the management server fail the reconnect with Unable to acquire lock on host while the convert was running.

B. virsh blockcommit fails before the job starts, and the failure is reported as success.
Script.runScript() returns null when execute() returns an error, so mergeSnapshotIntoBaseFileWithoutEvents()
gets null for any non-zero exit and logs Successfully merged snapshot. The snapshot ends BackedUp.
Reproduced with:

  • a NAS backup (backup-begin) started during the convert and still running at merge time:
    Operation not supported: disk 'vdb' already in active block job;
  • the domain job held by another operation for more than 30 s:
    Timed out during operation: cannot acquire state change lock.

NAS backups do not go through the VM work queue and do not check for volumes in Snapshotting, so nothing prevents
this overlap.

versions

ACS 4.20+

The steps to reproduce the bug
  1. Running VM with a data disk on NFS, write some data so the convert takes a while.
  2. Take a volume snapshot of the data disk.
  3. As soon as virsh domblklist shows the disk on a new bare-UUID file, either:
    • kill -9 the agent JVM; or
    • start a backup: virsh backup-begin <vm> (push or pull mode) and keep it running until the merge.
  4. Write data in the guest, then stop and start the VM from CloudStack.
What to do about it?

Expected:
The merge either completes, or the snapshot fails and the VM is not left on an overlay that CloudStack does not track.

Actual:
The VM runs on /mnt/<pool>/<uuid> (backing file = the volume). Case B logs:

WARN  Process [N] for command [/bin/bash -c virsh blockcommit <vm> vdb --base ... --pivot ] encountered the error: [error: Operation not supported: disk 'vdb' already in active block job].
DEBUG Successfully merged snapshot [<overlay uuid>] into VM [<vm>] ...

Later snapshots of the volume fail with VM [...] has no disk with path [<base>].
After stop/start the guest reads the data from the time the snapshot started.

Proposed fix

  1. Detect merge failures. Make Script.runScript() (or at least the merge call) return an error on a non-zero
    exit. After the merge, check that the domain disk source is the base path again. If the disk has another block
    job, wait and retry the commit. If it still fails, fail the snapshot with a clear error instead of BackedUp.
  2. Recover after agent death. Before creating the overlay, write a marker (vm, disk, base, overlay) on the shared primary pool, keyed by volume. On every VM start, and on agent start once the pools are mounted, any host that finds a marker reconciles the volume:
  • Domain still running on the overlay: run blockcommit --active --pivot, or pivot the commit job that is already running.
  • Domain gone: commit the overlay offline with qemu-img commit, then start.
  • Overlay still locked by a live qemu: refuse to start.
  • Base written since the marker: never commit, raise an alert and keep the overlay.
    Remove the marker only after domblklist shows the base path again.
  1. Do not exit with commands in progress. On a ReadyCommand with details, reconnect (as isExitOnFailures()=false
    already does) instead of System.exit(1), at least while commands are in progress.
  2. Serialize NAS backups with volume snapshots of the same VM (VM work queue or refuse while a volume is
    Snapshotting).
Ngôn ngữ chính
Java
Star
3.1k
Fork
1.4k
Merge trung bình
7 ngày 16 giờ
Pull request đã merge (30 ngày)
15

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/cloudstack

Tất cả issue của apache/cloudstack

Issue tương tự

Thêm issue về Java

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.