StorPool: resize of an encrypted volume does not account for the LUKS header (live resize fails with "Cannot grow device files", guest ends up 2 MiB short)
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 55/100
Research direction
Start in plugins/storage/volume/storpool/.../StorPoolPrimaryDataStoreDriver.java: the create path adds the 2097152-byte LUKS allowance (around L302, L876, L879), while updateVolumeByOffering, updateVolumeByStorPoolQoS and the rollback in resizeVolume (around L420-512) do not. Also read resizevolume.sh, where notifyqemu logs but still exits 0. Done means encrypted resizes send newSize + 2097152 to StorPool, the rollback uses the same adjusted size, the agent and database keep the plain size, and a test covers the encrypted case.
Written by the indexing model from the issue text.
Description
problem
When a LUKS-encrypted volume on StorPool primary storage is resized, the
plugin resizes the StorPool volume to exactly the requested size. At creation
the plugin adds 2 MiB (2097152 bytes) for the LUKS header, but the resize path
does not.
Effects:
-
Live resize fails inside QEMU. The agent runs
virsh blockresizewith the
new size; QEMU needs new size + LUKS payload offset on the underlying block
device, which is larger than the device, and returns:internal error: unable to execute QEMU command 'block_resize': Cannot grow device filesThe failure is not surfaced.
notifyqemu()inresizevolume.shonly logs
"failed to live resize", the script exits 0, the agent returns success and
the resizeVolume job succeeds. The guest keeps seeing the old size. -
Size mismatch. CloudStack records the requested size, but once QEMU
re-reads the device (VM stop/start, or a manual blockresize) the guest sees
the requested size minus the LUKS header. The same shortfall applies to
offline resize, without any error (from code reading, not tested). -
Rollback uses the same unadjusted size (from code reading, not tested).
If the StorPool API call or the agent command fails,resizeVolume()calls
StorPoolUtil.volumeUpdate(name, oldSize, true, ...), i.e. shrinkOk=true
with the CloudStack size. For an encrypted volume that has never been
resized the device is oldSize + 2097152, so this requests a 2 MiB shrink of
a LUKS device.
Code references (main at cc30155defc89bce24a253df892a0494ae8bba1e):
- Create adds the allowance:
https://github.com/apache/cloudstack/blob/cc30155defc89bce24a253df892a0494ae8bba1e/plugins/storage/volume/storpool/src/main/java/org/apache/cloudstack/storage/datastore/driver/StorPoolPrimaryDataStoreDriver.java#L302
(also L876 and L879 for volumes created from a template/snapshot) - Resize does not (
updateVolumeByStorPoolQoS,updateVolumeByOffering):
https://github.com/apache/cloudstack/blob/cc30155defc89bce24a253df892a0494ae8bba1e/plugins/storage/volume/storpool/src/main/java/org/apache/cloudstack/storage/datastore/driver/StorPoolPrimaryDataStoreDriver.java#L495-L512 - Rollback with oldSize and shrinkOk=true:
https://github.com/apache/cloudstack/blob/cc30155defc89bce24a253df892a0494ae8bba1e/plugins/storage/volume/storpool/src/main/java/org/apache/cloudstack/storage/datastore/driver/StorPoolPrimaryDataStoreDriver.java#L420-L436 - QEMU side:
block_crypto_co_truncateadds the payload offset, and
raw_co_truncaterejects growth of a block device:
https://github.com/qemu/qemu/blob/v9.0.0/block/crypto.c#L433-L449
https://github.com/qemu/qemu/blob/v9.0.0/block/file-posix.c#L2670-L2678
The header allowance was introduced with StorPool encryption support in #7539.
versions
on all versions
The steps to reproduce the bug
- KVM zone with StorPool primary storage and a host that supports volume
encryption. Create a custom-size disk offering with encrypt=true, tagged
to the StorPool pool. - Create a 10 GiB data volume from that offering and attach it to a running
VM. On the host,blockdev --getsize64on the StorPool device shows
10739515392 (10 GiB + 2097152); the guest sees 10737418240. - Resize the volume while the VM is running:
cmk resize volume id= size=20 - Observe:
- the resizeVolume job succeeds and CloudStack reports 20 GiB
blockdev --getsize64shows 21474836480 (exactly 20 GiB, no allowance)- libvirtd logs the block_resize "Cannot grow device files" error
- the guest still sees 10 GiB
Expected: the StorPool volume is resized to 20 GiB + 2097152, blockresize
succeeds, and the guest sees 20 GiB, matching what CloudStack reports.
What to do about it?
Apply the same allowance on resize as on create, for encrypted volumes only:
- In
updateVolumeByOfferingandupdateVolumeByStorPoolQoS, send
payload.newSize + 2097152to the StorPool API when the volume has a
passphrase (or KMS key on main).updateVolumeByStorPoolQoSneeds the
volume passed in. - Use the same adjusted size for the rollback in
resizeVolume(). - Keep
payload.newSizefor the agent command and for the size stored in
the database.
Sketch:
private static long getStorPoolVolumeSize(VolumeObject vol, long size) {
boolean encrypted = vol.getPassphraseId() != null || vol.getKmsKeyId() != null;
return encrypted ? size + 2097152 : size;
}
Volumes already resized under this bug have no allowance; the next resize
after the fix would correct them.
Separately, it may be worth surfacing the blockresize failure instead of
returning success, so a failed live resize is visible in the job result.
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.4k
- Avg merge
- 7d 4h
- Merged PRs (30d)
- 13
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/cloudstack
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/cloudstack#14719 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
apache/cloudstack#14715 ·
Maintainers usually reply within 1 day
-
resizeVolume of a running system VM's volume is accepted but the VM never sees the new sizePossibly taken @bhouse-nexthop claimed this 7 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/cloudstack#14645 · 1 comment ·
Maintainers usually reply within 1 day
-
VR reports a load balancer change as applied when haproxy.cfg cannot be replacedPossibly taken @bhouse-nexthop claimed this 7 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/cloudstack#14633 · 1 comment ·
Maintainers usually reply within 1 day
-
restartNetwork reports a ResourceUnavailableException as an unexpected error (530), unlike restartVPC (534)Possibly taken @bhouse-nexthop claimed this 7 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/cloudstack#14631 · 1 comment ·
Maintainers usually reply within 1 day
All issues in apache/cloudstack
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
SAP/ai-sdk-java#1206 ·
Maintainers usually reply within 1 day
-
enhancement good first issue insight-module
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
sdlc-pro/spring-lens#606 ·
Maintainers usually reply within 1 day
-
quality-assurance
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Paul-Austin-Oswego-CSC480-HCI521/gift-app#252 ·
Maintainers usually reply within 1 day
-
backlog kind/bug
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
TencentBlueKing/bk-job#4627 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 63/100
openrocket/openrocket#3460 ·
Maintainers usually reply within 1 day