Snapshots copy from Linstor to NFS fail
Assessment
This issue has not been assessed yet.
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
Storage - Linstor snapshot
CLOUDSTACK VERSION
4.22.1.0
CONFIGURATION
N/A
OS / ENVIRONMENT
- Cloudstack 4.22.1.0
- Hypervisor KVM
- AlmaLinux 9.8
- Primary storage: Linstor (LVM-thin)
- Secondary storage: NFS
SUMMARY
createSnapshot on any volume backed by a Linstor primary storage pool fails 100% of the time with: Failed to create Snapshot due to an internal error creating Snapshot for volume <volume-uuid>.
The failure is deterministic and reproducible on every attempt, regardless of volume, VM, or snapshot name.
First the snapshot is created succesfully on the Linstor backend, but then the copy to NFS (secundairy storage) fails as CloudStack's KVM agent invokes qemu-img convert against a /dev/mapper/... device path that does not match the path the kernel actually created.
Actual device location: /dev/mapper/linstor_pool-lv--thin-cs--12fc4055--3985--4025--8eb5--d6fd53effe37_00000
Path CloudStack looks for: /dev/mapper/linstor_pool-lvm-thin-cs--12fc4055--3985--4025--8eb5--d6fd53effe37_00000_cs--d7aea646--5f40--46a2--b9dc--77e41ea29336
Notice the missing dashes in the linstor_pool-lv--thin-cs-name.
STEPS TO REPRODUCE
Create disk snapshot on a Linstor volume with NFS as secundairy storage and observe the error message.
EXPECTED RESULTS
Snapshot is created on the Linstor backend and successfully copied to secondary (NFS) storage.
ACTUAL RESULTS
The error logged in management-server.log:
ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] Failed to create Snapshot due to an internal error creating Snapshot for volume 12fc4055-3985-4025-8eb5-d6fd53effe37 java.lang.RuntimeException: Unexpected exception
at com.cloud.storage.VolumeApiServiceImpl.takeSnapshotInternal(VolumeApiServiceImpl.java:3999)
at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:3910)
...
Caused by: com.cloud.utils.exception.CloudRuntimeException: Exception: org.apache.cloudstack.utils.qemu.QemuImgException
Message: qemu-img: Could not open '/dev/mapper/linstor_pool-lvm-thin-cs--12fc4055--3985--4025--8eb5--d6fd53effe37_00000_cs--d7aea646--5f40--46a2--b9dc--77e41ea29336': Could not open '/dev/mapper/linstor_pool-lvm-thin-cs--12fc4055--3985--4025--8eb5--d6fd53effe37_00000_cs--d7aea646--5f40--46a2--b9dc--77e41ea29336': No such file or directory
Stack: org.apache.cloudstack.utils.qemu.QemuImgException: ...
at org.apache.cloudstack.utils.qemu.QemuImg.convert(QemuImg.java:495)
...
at com.cloud.hypervisor.kvm.resource.wrapper.LinstorBackupSnapshotCommandWrapper.convertImageToQCow2(LinstorBackupSnapshotCommandWrapper.java:99)
at com.cloud.hypervisor.kvm.resource.wrapper.LinstorBackupSnapshotCommandWrapper.execute(LinstorBackupSnapshotCommandWrapper.java:156)
...
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:475)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:204)
at com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1787)
at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1660)
...
Suggested fix: apply LVM's standard dash-doubling escape to the volume-group name the same way it's already correctly applied to the cs--<uuid> logical-volume/snapshot name portion.
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.4k
- Avg merge
- 6d 20h
- Merged PRs (30d)
- 27
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/cloudstack
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 90/100
apache/cloudstack#14222 ·
-
bug component:kubernetes
Difficulty 1/5 Under an hour Newbie friendliness 88/100
apache/cloudstack#14180 ·
-
bug component:projects component:UI
Difficulty 1/5 Under an hour Newbie friendliness 88/100
apache/cloudstack#14070 · 5 comments ·
-
component:backup
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/cloudstack#14013 ·
-
KVM agent fails to connect to Ceph RBD storage pool after upgrading Ceph client to Tentacle 20.2.4 Openbug component:ceph
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/cloudstack#13989 · 3 comments ·
All issues in apache/cloudstack
Similar issues
-
area/plugin
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
kestra-io/plugin-kestra#190 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
google-ai-edge/LiteRT-LM#3739 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
integra-team-red/meet-map#249 ·
-
[Studio][Bug] Cancelled create-user dialog keeps the password and admin switch for the next attempt Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
apache/rocketmq-dashboard#5064 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
wso2/dpdp-accelerator#287 ·