Not able to delete CLVM_NG primary storage (Failed to delete storage pool on host)

Start by tracing the deleteStoragePool API request through DeleteStoragePoolCommand and CloudStackPrimaryDataStoreLifeCycleImpl, using the reported libvirt error that the storage pool is not active. Review how CLVM_NG pools are handled across the listed hosts and reproduce the deletion failure if possible. Done means an empty CLVM_NG primary storage can be deleted without the storage-pool failure.

Description:

Not able to delete CLVM_NG primary storage. I am getting "Failed to delete storage pool on host" error message.

All VMs and associated volumes, templates, etc were deleted from primary storage and it was put under maintenance state.

This may be related to: https://github.com/apache/cloudstack/discussions/13392

# vgs
  VG			  #PV #LV #SN Attr   VSize    VFree
  hdd_raid6_vg    1   1   0 wz--n-   <4.37t       0
  rhel_vg         1   6   0 wz--n- <444.61g <100.61g
  ssd_raid1_vg    1   1   0 wz--n-    3.49t       0
  ds_test_01_vg   1   0   0 wz--ns   <4.00t   <4.00t
  ds_test_02_vg   1   0   0 wz--ns   <4.00t   <4.00t
  ds_test_03_vg   1   0   0 wz--ns   <4.00t   <4.00t
  ds_test_04_vg   1   0   0 wz--ns   <4.00t   <4.00t
  ds_test_05_vg   1   0   0 wz--ns   <4.00t   <4.00t
#

Workaround:

Removed all shared VGs and restarted agent service on all nodes

# vgs
  VG           #PV #LV #SN Attr   VSize    VFree
  hdd_raid6_vg   1   1   0 wz--n-   <4.37t       0
  rhel_vg        1   6   0 wz--n- <444.61g <100.61g
  ssd_raid1_vg   1   1   0 wz--n-    3.49t       0

# /usr/bin/systemctl restart cloudstack-agent.service
#

Logs from ACS server:

2026-09-18 13:40:56,985 DEBUG [c.c.a.ApiServlet] (qtp1390913202-47871:[ctx-90bdde62]) (logid:ac4f0fba) ===START===  108.145.117.211 -- POST
command=deleteStoragePool
response=json
id=0b35ce6c-7826-3188-b0bb-1150c455a5a3
forced=true
sessionkey=Ln4wx0lPZhcXCec2jVuN6FNC5Sk
2026-09-18 13:40:56,985 DEBUG [c.c.a.ApiServlet] (qtp1390913202-47871:[ctx-90bdde62]) (logid:ac4f0fba) Two factor authentication is already verified for the user 2, so skipping
2026-09-18 13:40:56,988 DEBUG [c.c.a.ApiServer] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"1b4fa8b0-8f51-11f1-ba8c-525400083ffc"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2026-09-18 13:40:56,989 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Account [Account [{"accountName":"admin","id":2,"uuid":"1b4fa8b0-8f51-11f1-ba8c-525400083ffc"}]] is Root Admin and there aren't any API key pair permissions involved, thus, all APIs are allowed.
2026-09-18 13:40:56,989 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-09-18 13:40:56,989 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-09-18 13:40:57,003 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
2026-09-18 13:40:57,004 DEBUG [c.c.a.m.ClusteredAgentAttache] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 61-5210101818914280197: Routed from 90520734213782
2026-09-18 13:40:57,004 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 21-5210101818914280197: Sending  { Cmd , MgmtId: 90520734213782, via: 21(esx0101), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"74","uuid":"0b35ce6c-7826-3188-b0bb-1150c455a5a3","host":"localhost","path":"ds_test_01_vg","port":"0","type":"CLVM_NG"},"_localPath":"/mnt//4df4b702-17ad-35ae-8003-1286a2ffb702","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-18 13:40:57,047 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 21-5210101818914280197: Received:  { Ans: , MgmtId: 90520734213782, via: 21(esx0101), Ver: v1, Flags: 10, { Answer } }
2026-09-18 13:40:57,047 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Details from executing class com.cloud.agent.api.DeleteStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Requested operation is not valid: storage pool 'ds_test_01_vg' is not active
2026-09-18 13:40:57,047 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Failed to delete storage pool: false
2026-09-18 13:40:57,047 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
2026-09-18 13:40:57,048 DEBUG [c.c.a.m.ClusteredAgentAttache] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 61-1204994375298338227: Routed from 90520734213782
2026-09-18 13:40:57,048 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 22-1204994375298338227: Sending  { Cmd , MgmtId: 90520734213782, via: 22(esx0102), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"74","uuid":"0b35ce6c-7826-3188-b0bb-1150c455a5a3","host":"localhost","path":"ds_test_01_vg","port":"0","type":"CLVM_NG"},"_localPath":"/mnt//4df4b702-17ad-35ae-8003-1286a2ffb702","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-18 13:40:57,050 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 22-1204994375298338227: Received:  { Ans: , MgmtId: 90520734213782, via: 22(esx0102), Ver: v1, Flags: 10, { Answer } }
2026-09-18 13:40:57,050 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Details from executing class com.cloud.agent.api.DeleteStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Requested operation is not valid: storage pool 'ds_test_01_vg' is not active
2026-09-18 13:40:57,050 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Failed to delete storage pool: false
2026-09-18 13:40:57,051 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
2026-09-18 13:40:57,051 DEBUG [c.c.a.m.ClusteredAgentAttache] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 61-6906832978526096897: Routed from 90520734213782
2026-09-18 13:40:57,051 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 23-6906832978526096897: Sending  { Cmd , MgmtId: 90520734213782, via: 23(esx0103), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"74","uuid":"0b35ce6c-7826-3188-b0bb-1150c455a5a3","host":"localhost","path":"ds_test_01_vg","port":"0","type":"CLVM_NG"},"_localPath":"/mnt//4df4b702-17ad-35ae-8003-1286a2ffb702","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-18 13:40:57,054 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 23-6906832978526096897: Received:  { Ans: , MgmtId: 90520734213782, via: 23(esx0103), Ver: v1, Flags: 10, { Answer } }
2026-09-18 13:40:57,054 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Details from executing class com.cloud.agent.api.DeleteStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Requested operation is not valid: storage pool 'ds_test_01_vg' is not active
2026-09-18 13:40:57,054 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Failed to delete storage pool: false
2026-09-18 13:40:57,054 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
2026-09-18 13:40:57,055 DEBUG [c.c.a.m.ClusteredAgentAttache] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 61-5033617008516681232: Routed from 90520734213782
2026-09-18 13:40:57,055 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 24-5033617008516681232: Sending  { Cmd , MgmtId: 90520734213782, via: 24(esx0104), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"74","uuid":"0b35ce6c-7826-3188-b0bb-1150c455a5a3","host":"localhost","path":"ds_test_01_vg","port":"0","type":"CLVM_NG"},"_localPath":"/mnt//4df4b702-17ad-35ae-8003-1286a2ffb702","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-18 13:40:57,057 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 24-5033617008516681232: Received:  { Ans: , MgmtId: 90520734213782, via: 24(esx0104), Ver: v1, Flags: 10, { Answer } }
2026-09-18 13:40:57,057 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Details from executing class com.cloud.agent.api.DeleteStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Requested operation is not valid: storage pool 'ds_test_01_vg' is not active
2026-09-18 13:40:57,057 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Failed to delete storage pool: false
2026-09-18 13:40:57,058 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
2026-09-18 13:40:57,058 DEBUG [c.c.a.m.ClusteredAgentAttache] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 61-1866460570568379857: Routed from 90520734213782
2026-09-18 13:40:57,058 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 25-1866460570568379857: Sending  { Cmd , MgmtId: 90520734213782, via: 25(esx0307), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"74","uuid":"0b35ce6c-7826-3188-b0bb-1150c455a5a3","host":"localhost","path":"ds_test_01_vg","port":"0","type":"CLVM_NG"},"_localPath":"/mnt//4df4b702-17ad-35ae-8003-1286a2ffb702","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-18 13:40:57,101 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 25-1866460570568379857: Received:  { Ans: , MgmtId: 90520734213782, via: 25(esx0307), Ver: v1, Flags: 10, { Answer } }
2026-09-18 13:40:57,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Details from executing class com.cloud.agent.api.DeleteStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Requested operation is not valid: storage pool 'ds_test_01_vg' is not active
2026-09-18 13:40:57,101 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Failed to delete storage pool: false
2026-09-18 13:40:57,101 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
2026-09-18 13:40:57,101 DEBUG [c.c.a.m.ClusteredAgentAttache] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 61-971088669651783173: Routed from 90520734213782
2026-09-18 13:40:57,102 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 26-971088669651783173: Sending  { Cmd , MgmtId: 90520734213782, via: 26(esx0308), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"74","uuid":"0b35ce6c-7826-3188-b0bb-1150c455a5a3","host":"localhost","path":"ds_test_01_vg","port":"0","type":"CLVM_NG"},"_localPath":"/mnt//4df4b702-17ad-35ae-8003-1286a2ffb702","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-18 13:40:57,145 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 26-971088669651783173: Received:  { Ans: , MgmtId: 90520734213782, via: 26(esx0308), Ver: v1, Flags: 10, { Answer } }
2026-09-18 13:40:57,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Details from executing class com.cloud.agent.api.DeleteStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Requested operation is not valid: storage pool 'ds_test_01_vg' is not active
2026-09-18 13:40:57,145 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Failed to delete storage pool: false
2026-09-18 13:40:57,145 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
2026-09-18 13:40:57,146 DEBUG [c.c.a.m.ClusteredAgentAttache] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 61-3703084793605410297: Routed from 90520734213782
2026-09-18 13:40:57,146 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 27-3703084793605410297: Sending  { Cmd , MgmtId: 90520734213782, via: 27(esx0401), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"74","uuid":"0b35ce6c-7826-3188-b0bb-1150c455a5a3","host":"localhost","path":"ds_test_01_vg","port":"0","type":"CLVM_NG"},"_localPath":"/mnt//4df4b702-17ad-35ae-8003-1286a2ffb702","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-18 13:40:57,148 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 27-3703084793605410297: Received:  { Ans: , MgmtId: 90520734213782, via: 27(esx0401), Ver: v1, Flags: 10, { Answer } }
2026-09-18 13:40:57,148 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Details from executing class com.cloud.agent.api.DeleteStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Requested operation is not valid: storage pool 'ds_test_01_vg' is not active
2026-09-18 13:40:57,148 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Failed to delete storage pool: false
2026-09-18 13:40:57,149 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
2026-09-18 13:40:57,149 DEBUG [c.c.a.m.ClusteredAgentAttache] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 61-385902193070329372: Routed from 90520734213782
2026-09-18 13:40:57,149 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 28-385902193070329372: Sending  { Cmd , MgmtId: 90520734213782, via: 28(esx0402), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"74","uuid":"0b35ce6c-7826-3188-b0bb-1150c455a5a3","host":"localhost","path":"ds_test_01_vg","port":"0","type":"CLVM_NG"},"_localPath":"/mnt//4df4b702-17ad-35ae-8003-1286a2ffb702","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-18 13:40:57,193 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 28-385902193070329372: Received:  { Ans: , MgmtId: 90520734213782, via: 28(esx0402), Ver: v1, Flags: 10, { Answer } }
2026-09-18 13:40:57,193 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Details from executing class com.cloud.agent.api.DeleteStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Requested operation is not valid: storage pool 'ds_test_01_vg' is not active
2026-09-18 13:40:57,193 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Failed to delete storage pool: false
2026-09-18 13:40:57,193 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Wait time setting on com.cloud.agent.api.DeleteStoragePoolCommand is 1800 seconds
2026-09-18 13:40:57,194 DEBUG [c.c.a.m.ClusteredAgentAttache] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 61-4814910951612501578: Routed from 90520734213782
2026-09-18 13:40:57,194 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 29-4814910951612501578: Sending  { Cmd , MgmtId: 90520734213782, via: 29(esx0403), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"74","uuid":"0b35ce6c-7826-3188-b0bb-1150c455a5a3","host":"localhost","path":"ds_test_01_vg","port":"0","type":"CLVM_NG"},"_localPath":"/mnt//4df4b702-17ad-35ae-8003-1286a2ffb702","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-18 13:40:57,237 DEBUG [c.c.a.t.Request] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Seq 29-4814910951612501578: Received:  { Ans: , MgmtId: 90520734213782, via: 29(esx0403), Ver: v1, Flags: 10, { Answer } }
2026-09-18 13:40:57,237 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Details from executing class com.cloud.agent.api.DeleteStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Requested operation is not valid: storage pool 'ds_test_01_vg' is not active
2026-09-18 13:40:57,237 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) Failed to delete storage pool: false
2026-09-18 13:40:57,237 ERROR [c.c.a.ApiServer] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) unhandled exception executing api command: [Ljava.lang.String;@112cb1cd com.cloud.utils.exception.CloudRuntimeException: Failed to delete storage pool on host
        at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.deleteDataStore(CloudStackPrimaryDataStoreLifeCycleImpl.java:520)
        at com.cloud.storage.StorageManagerImpl.deleteDataStoreInternal(StorageManagerImpl.java:1808)
        at com.cloud.storage.StorageManagerImpl.deletePool(StorageManagerImpl.java:1711)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy140.deletePool(Unknown Source)
        at org.apache.cloudstack.api.command.admin.storage.DeletePoolCmd.execute(DeletePoolCmd.java:71)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:820)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:705)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:414)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:191)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:188)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:147)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:840)
2026-09-18 13:40:57,238 DEBUG [c.c.a.ApiServlet] (qtp1390913202-47871:[ctx-90bdde62, ctx-9aa46af6]) (logid:ac4f0fba) ===END===  108.145.117.211 -- POST
command=deleteStoragePool
response=json
id=0b35ce6c-7826-3188-b0bb-1150c455a5a3
forced=true
sessionkey=Ln4wx0lPZhcXCec2jVuN6FNC5Sk
2026-09-18 13:40:57,701 INFO  [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-5d77e301]) (logid:c0016586) Found 1 inactive management server node based on timestamp

Logs from agent:

2026-09-18 14:08:38,906 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:f7ddf00d) Request:Seq 21-5210101818914280260:  { Cmd , MgmtId: 90520734213782, via: 21, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.DeleteStoragePoolCommand":{"_pool":{"id":"74","uuid":"0b35ce6c-7826-3188-b0bb-1150c455a5a3","host":"localhost","path":"ds_test_01_vg","port":"0","type":"CLVM_NG"},"_localPath":"/mnt//4df4b702-17ad-35ae-8003-1286a2ffb702","_removeDatastore":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-09-18 14:08:38,906 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:f7ddf00d) Processing command: com.cloud.agent.api.DeleteStoragePoolCommand
2026-09-18 14:08:38,906 INFO  [kvm.storage.ClvmStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:f7ddf00d) Attempting to remove storage pool 0b35ce6c-7826-3188-b0bb-1150c455a5a3 from libvirt
2026-09-18 14:08:38,906 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:f7ddf00d) Looking for libvirtd connection at: qemu:///system
2026-09-18 14:08:38,907 INFO  [kvm.storage.ClvmStorageAdaptor] (AgentRequest-Handler-5:[]) (logid:f7ddf00d) Storage pool 0b35ce6c-7826-3188-b0bb-1150c455a5a3 has no corresponding secret. Not removing any secret.

versions

ACS 4.23
KVM hosts
CLVM_NG primary storage