VM migration fails with "Domain not found" and ambiguous libvirt connection handling

Bắt đầu với đường dẫn MigrateCommand trong agent KVM/libvirt và so sánh nó với các log migration của management-server được cung cấp trong issue. Theo dõi cách source host phân giải VM domain và cách các lỗi được báo cáo; hoàn thành khi migration không còn thất bại với "Domain not found" và management-server hiển thị lỗi liên quan.

problem

We are experiencing a VM migration failure in CloudStack where the migration process fails with a "Domain not found" error on the source host.

During migration, CloudStack reports "Migration was unsuccessful" without detailed error information on the management server. However, on the source host (KVM/libvirt), the agent fails to locate the VM domain during execution of the MigrateCommand.

**log management-server
2026-04-21 15:21:29,314 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp652176954-26811:[ctx-82bf6948, ctx-c49af74b]) (logid:8f7aa9da) submit async job-303869, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd","cmdInfo":"{\"virtualmachineid\":\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\",\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"xxx\",\"hostid\":\"2995de01-4d98-49c9-b379-ff8adf0f0722\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"6185591\",\"ctxDetails\":\"{\\\"interface com.cloud.host.Host\\\":\\\"2995de01-4d98-49c9-b379-ff8adf0f0722\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\\\"}\",\"ctxAccountId\":\"2\",\"cmdEventType\":\"VM.MIGRATE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":303869,"initMsid":345050754440,"instanceId":4127,"instanceType":"VirtualMachine","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"569f38b1-bc6c-422c-a4ee-5fc8e3ebb7c3"} 2026-04-21 15:21:29,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-57:[ctx-964ae7d1, job-303869]) (logid:569f38b1) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd","cmdInfo":"{\"virtualmachineid\":\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\",\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"xxx\",\"hostid\":\"2995de01-4d98-49c9-b379-ff8adf0f0722\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"6185591\",\"ctxDetails\":\"{\\\"interface com.cloud.host.Host\\\":\\\"2995de01-4d98-49c9-b379-ff8adf0f0722\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"bf5f467f-c17a-4e6e-b3e2-1beb97941324\\\"}\",\"ctxAccountId\":\"2\",\"cmdEventType\":\"VM.MIGRATE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":303869,"initMsid":345050754440,"instanceId":4127,"instanceType":"VirtualMachine","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"569f38b1-bc6c-422c-a4ee-5fc8e3ebb7c3"} 2026-04-21 15:21:29,353 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1) Checking if there are any ongoing snapshots volumes associated with VM VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} 2026-04-21 15:21:29,354 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1) Found 1 no. of volumes of type null for vm with VM ID VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} 2026-04-21 15:21:29,355 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1) The count of ongoing Snapshots for VM VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} and disk type null is 0 2026-04-21 15:21:29,355 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-57:[ctx-964ae7d1, job-303869, ctx-ff0b825e]) (logid:569f38b1) Found no ongoing snapshots on volumes associated with the vm VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} 2026-04-21 15:21:30,999 INFO [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Migrating VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} to Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(2)-Pod(2)-Cluster(6)-Host(324)-Storage()] 2026-04-21 15:21:31,022 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Preparing to migrate [1] volumes for VM [VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}]. 2026-04-21 15:21:31,106 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Seq 446-3275242829005214246: Forwarding Seq 324-3275242829005214246: { Cmd , MgmtId: 345050754440, via: 324(com11), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.PrepareForMigrationCommand":{"vm":{"id":"4127","name":"i-2-4127-VM","state":"Running","type":"User","cpus":"2","minSpeed":"400","maxSpeed":"2000","minRam":"(2.00 GB) 2147483648","maxRam":"(2.00 GB) 2147483648","arch":"x86_64","os":"Other Ubuntu (64-bit)","platformEmulator":"Other Ubuntu","bootArgs":"","enableHA":"true","limitCpuUse":"false","enableDynamicallyScaleVm":"true","details":{"cpuOvercommitRatio":"5.0","rootdisksize":"20","rootDiskController":"osdefault"},"uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c9e66d60-139c-4e90-ab9a-f06d6486b63d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","name":"SSD","id":"66","poolType":"StorPool","host":"n/a","path":"/dev/storpool-byid/","port":"0","url":"StorPool://n/a/dev/storpool-byid//?ROLE=Primary&STOREUUID=ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","isManaged":"false"}},"name":"ROOT-4127","size":"(20.00 GB) 21474836480","path":"/dev/storpool-byid/<volume-id>","volumeId":"15599","vmName":"i-2-4127-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"66","id":"15599","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"true"}},"diskSeq":"0","path":"/dev/storpool-byid/<volume-id>","type":"ROOT","details":{"storageHost":"n/a","managed":"false","storagePort":"0","protocoltype":"StorPool","storage.pool.disk.wait":"60","volumeSize":"(20.00 GB) 21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"false","nicUuid":"b1aee509-ad6b-4dd5-ade5-c1d44f995615","details":{"MacLearning":"false","PromiscuousMode":"false","internetProtocol":"IPv4","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","networkId":"783","networkSegmentName":"D1-A2-Z2-V1124-S783","uuid":"958d365a-d975-419f-830a-b13b7f29389d","ip":"192.168.10.125","netmask":"255.255.255.0","gateway":"192.168.10.1","mac":"02:02:03:0f:00:3c","dns1":"8.8.8.8","broadcastType":"Vxlan","type":"Guest","broadcastUri":"vxlan://17742","isolationUri":"vxlan://17742","securityGroupEnabled":"false","name":"cloudbr2"}],"vcpuMaxLimit":"2","configDriveLabel":"config-2","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{"783":"D1-A2-Z2-V1124-S783"}},"rollback":"false","wait":"0","bypassHostMaintenance":"false"}}] } to 345050719314 2026-04-21 15:21:31,368 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} state transited from [Running] to [Migrating] with event [MigrationRequested]. VM's original host: Host {"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"}, new host: Host {"id":324,"name":"com11","type":"Routing","uuid":"2995de01-4d98-49c9-b379-ff8adf0f0722"}, host before state transition: Host {"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"} 2026-04-21 15:21:31,396 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Setting VLAN persistence to [{"17742":true}] as part of migrate command for VM [VM {id: "4127", name: "i-2-4127-VM", uuid: "bf5f467f-c17a-4e6e-b3e2-1beb97941324", type: "User"}]. 2026-04-21 15:21:31,396 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Setting CPU shares to [800] as part of migrate command for VM [VM {id: "4127", name: "i-2-4127-VM", uuid: "bf5f467f-c17a-4e6e-b3e2-1beb97941324", type: "User"}]. 2026-04-21 15:21:31,399 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Seq 446-8848728842852947950: Forwarding Seq 318-8848728842852947950: { Cmd , MgmtId: 345050754440, via: 318(com10), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.MigrateCommand":{"vmName":"i-2-4127-VM","destinationIp":"10.xx.xx.61","migrateStorageManaged":"false","migrateNonSharedInc":"false","autoConvergence":"false","hostGuid":"dca8e648-8706-3504-adb2-7f20ea59398c-LibvirtComputingResource","windows":"false","virtualMachine":{"id":"4127","name":"i-2-4127-VM","state":"Running","type":"User","cpus":"2","minSpeed":"400","maxSpeed":"2000","minRam":"(2.00 GB) 2147483648","maxRam":"(2.00 GB) 2147483648","arch":"x86_64","os":"Other Ubuntu (64-bit)","platformEmulator":"Other Ubuntu","bootArgs":"","enableHA":"true","limitCpuUse":"false","enableDynamicallyScaleVm":"true","details":{"cpuOvercommitRatio":"5.0","rootdisksize":"20","rootDiskController":"osdefault"},"uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c9e66d60-139c-4e90-ab9a-f06d6486b63d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","name":"SSD","id":"66","poolType":"StorPool","host":"n/a","path":"/dev/storpool-byid/","port":"0","url":"StorPool://n/a/dev/storpool-byid//?ROLE=Primary&STOREUUID=ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","isManaged":"false"}},"name":"ROOT-4127","size":"(20.00 GB) 21474836480","path":"/dev/storpool-byid/<volume-id>","volumeId":"15599","vmName":"i-2-4127-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"66","id":"15599","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"true"}},"diskSeq":"0","path":"/dev/storpool-byid/<volume-id>","type":"ROOT","details":{"storageHost":"n/a","managed":"false","storagePort":"0","protocoltype":"StorPool","storage.pool.disk.wait":"60","volumeSize":"(20.00 GB) 21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"false","nicUuid":"b1aee509-ad6b-4dd5-ade5-c1d44f995615","details":{"MacLearning":"false","PromiscuousMode":"false","internetProtocol":"IPv4","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","networkId":"783","networkSegmentName":"xxx","uuid":"958d365a-d975-419f-830a-b13b7f29389d","ip":"192.168.10.125","netmask":"255.255.255.0","gateway":"192.168.10.1","mac":"02:02:03:0f:00:3c","dns1":"8.8.8.8","broadcastType":"Vxlan","type":"Guest","broadcastUri":"vxlan://17742","isolationUri":"vxlan://17742","securityGroupEnabled":"false","name":"cloudbr2"}],"vcpuMaxLimit":"2","configDriveLabel":"config-2","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{"783":"D1-A2-Z2-V1124-S783"}},"executeInSequence":"false","migrateDiskInfoList":[],"dpdkInterfaceMapping":{},"newVmCpuShares":"800","vlanToPersistenceMap":{"17742":"true"},"wait":"0","bypassHostMaintenance":"false"}}] } to 345050719314 2026-04-21 15:21:31,447 INFO [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Migration was unsuccessful. Cleaning up: VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} 2026-04-21 15:21:31,454 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) Releasing [1] volumes for VM [VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"}] from host [Host {"id":324,"name":"com11","type":"Routing","uuid":"2995de01-4d98-49c9-b379-ff8adf0f0722"}]. 2026-04-21 15:21:31,463 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Migrating","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} is a [User], returning null for control Nic IP. 2026-04-21 15:21:31,482 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-16:[ctx-4b90bc2f, job-303869/job-303872, ctx-1d37bf07]) (logid:569f38b1) VM instance {"id":4127,"instanceName":"i-2-4127-VM","state":"Running","type":"User","uuid":"bf5f467f-c17a-4e6e-b3e2-1beb97941324"} state transited from [Migrating] to [Running] with event [OperationFailed]. VM's original host: Host {"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"}, new host: Host {"id":318,"name":"com10","type":"Routing","uuid":"c8b05f7c-09cc-467e-b626-74048a6a870b"}, host before state transition: Host {"id":324,"name":"com11","type":"Routing","uuid":"2995de01-4d98-49c9-b379-ff8adf0f0722"}

Additional Context

From the libvirt/qemu logs on the source host, it is observed that the VM domain did exist previously as part of an earlier migration.

At around 15:20, the VM was successfully started on the source host and participated in a migration process:

**log cloudstack-agent (source host)
2026-04-21 15:21:29,362 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:569f38b1) Processing command: com.cloud.agent.api.GetVmDiskStatsCommand 2026-04-21 15:21:29,362 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-3:[]) (logid:569f38b1) Looking for libvirtd connection at: qemu:///system 2026-04-21 15:21:29,374 WARN [resource.wrapper.LibvirtGetVmDiskStatsCommandWrapper] (AgentRequest-Handler-3:[]) (logid:569f38b1) Can't get vm disk stats: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-2-4127-VM', continue 2026-04-21 15:21:29,375 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:569f38b1) Seq 318-8848728842852947947: { Ans: , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 10, [{"com.cloud.agent.api.GetVmDiskStatsAnswer":{"hostName":"com10","vmDiskStatsMap":{},"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] } 2026-04-21 15:21:29,422 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:569f38b1) Processing command: com.cloud.agent.api.GetVmNetworkStatsCommand 2026-04-21 15:21:29,422 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:569f38b1) Looking for libvirtd connection at: qemu:///system 2026-04-21 15:21:29,435 WARN [resource.wrapper.LibvirtGetVmNetworkStatsCommandWrapper] (AgentRequest-Handler-2:[]) (logid:569f38b1) Can't get vm network stats: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-2-4127-VM', continue 2026-04-21 15:21:29,436 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:569f38b1) Seq 318-8848728842852947948: { Ans: , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 10, [{"com.cloud.agent.api.GetVmNetworkStatsAnswer":{"hostName":"com10","vmNetworkStatsMap":{},"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] } 2026-04-21 15:21:31,403 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:569f38b1) Request:Seq 318-8848728842852947950: { Cmd , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.MigrateCommand":{"vmName":"i-2-4127-VM","destinationIp":"10.120.25.61","migrateStorageManaged":"false","migrateNonSharedInc":"false","autoConvergence":"false","hostGuid":"dca8e648-8706-3504-adb2-7f20ea59398c-LibvirtComputingResource","windows":"false","virtualMachine":{"id":"4127","name":"i-2-4127-VM","state":"Running","type":"User","cpus":"2","minSpeed":"400","maxSpeed":"2000","minRam":"(2.00 GB) 2147483648","maxRam":"(2.00 GB) 2147483648","arch":"x86_64","os":"Other Ubuntu (64-bit)","platformEmulator":"Other Ubuntu","bootArgs":"","enableHA":"true","limitCpuUse":"false","enableDynamicallyScaleVm":"true","details":{"cpuOvercommitRatio":"5.0","rootdisksize":"20","rootDiskController":"osdefault"},"uuid":"xxxx-c17a-4e6e-b3e2-1beb97941324","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c9e66d60-139c-4e90-ab9a-f06d6486b63d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","name":"SSD","id":"66","poolType":"StorPool","host":"n/a","path":"/dev/storpool-byid/","port":"0","url":"StorPool://n/a/dev/storpool-byid//?ROLE=Primary&STOREUUID=ssd;36b6ca0e-eb0f-4d11-9491-937d592f36e9","isManaged":"false"}},"name":"ROOT-4127","size":"(20.00 GB) 21474836480","path":"/dev/storpool-byid/<volume-id>","volumeId":"15599","vmName":"i-2-4127-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"66","id":"15599","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"true"}},"diskSeq":"0","path":"/dev/storpool-byid/<volume-id>","type":"ROOT","details":{"storageHost":"n/a","managed":"false","storagePort":"0","protocoltype":"StorPool","storage.pool.disk.wait":"60","volumeSize":"(20.00 GB) 21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"false","nicUuid":"b1aee509-ad6b-4dd5-ade5-c1d44f995615","details":{"MacLearning":"false","PromiscuousMode":"false","internetProtocol":"IPv4","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":"false","networkId":"783","networkSegmentName":"D1-A2-Z2-V1124-S783","uuid":"958d365a-d975-419f-830a-b13b7f29389d","ip":"192.168.10.125","netmask":"255.255.255.0","gateway":"192.168.10.1","mac":"02:02:03:0f:00:3c","dns1":"8.8.8.8","broadcastType":"Vxlan","type":"Guest","broadcastUri":"vxlan://17742","isolationUri":"vxlan://17742","securityGroupEnabled":"false","name":"cloudbr2"}],"vcpuMaxLimit":"2","configDriveLabel":"config-2","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{"783":"D1-A2-Z2-V1124-S783"}},"executeInSequence":"false","migrateDiskInfoList":[],"dpdkInterfaceMapping":{},"newVmCpuShares":"800","vlanToPersistenceMap":{"17742":"true"},"wait":"0","bypassHostMaintenance":"false"}}] } 2026-04-21 15:21:31,403 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:569f38b1) Processing command: com.cloud.agent.api.MigrateCommand 2026-04-21 15:21:31,419 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[]) (logid:569f38b1) Trying to migrate VM [i-2-4127-VM] to destination host: [qemu+tls://10.120.25.61/system]. 2026-04-21 15:21:31,419 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-5:[]) (logid:569f38b1) Property [hypervisor.uri] has empty or null value. Using default value [null]. 2026-04-21 15:21:31,419 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Looking for libvirtd connection at: qemu:///system 2026-04-21 15:21:31,431 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Can not find KVM connection for Instance: i-2-4127-VM, continuing. 2026-04-21 15:21:31,432 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-5:[]) (logid:569f38b1) Property [hypervisor.uri] has empty or null value. Using default value [null]. 2026-04-21 15:21:31,432 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Looking for libvirtd connection at: lxc:/// 2026-04-21 15:21:31,432 INFO [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) No existing libvirtd connection found. Opening a new one 2026-04-21 15:21:31,432 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-5:[]) (logid:569f38b1) Property [libvirt.events.enabled] has empty or null value. Using default value [false]. 2026-04-21 15:21:31,432 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Libvirt event listening is disabled, not setting up event loop 2026-04-21 15:21:31,433 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Can not find LXC connection for Instance: i-2-4127-VM, continuing. 2026-04-21 15:21:31,433 WARN [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Can not find a connection for Instance i-2-4127-VM. Assuming the default connection. 2026-04-21 15:21:31,433 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:569f38b1) Looking for libvirtd connection at: qemu:///system 2026-04-21 15:21:31,445 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-5:[]) (logid:569f38b1) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-2-4127-VM' 2026-04-21 15:21:31,446 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-5:[]) (logid:569f38b1) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-2-4127-VM' 2026-04-21 15:21:31,446 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[]) (logid:569f38b1) Found domain with name [i-2-4127-VM]. Starting VM migration to host [qemu+tls://10.120.25.61/system]. 2026-04-21 15:21:31,446 ERROR [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-5:[]) (logid:569f38b1) Can't migrate domain [i-2-4127-VM] due to: [Domain not found: no domain with matching name 'i-2-4127-VM']. org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-2-4127-VM' 2026-04-21 15:21:31,447 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:569f38b1) Seq 318-8848728842852947950: { Ans: , MgmtId: 345050754440, via: 318, Ver: v1, Flags: 10, [{"com.cloud.agent.api.MigrateAnswer":{"result":"false","details":"Domain not found: no domain with matching name 'i-2-4127-VM'","wait":"0","bypassHostMaintenance":"false"}}] }

**log libvirt (source host)
2026-04-21 08:20:25.095+0000: starting up libvirt version: 8.0.0, package: 23.2.module_el8.10.0+3869+b8959270 (AlmaLinux Packaging Team <packager@almalinux.org>, 2024-07-10-04:57:33, ), qemu version: 6.2.0qemu-kvm-6.2.0-53.module_el8.10.0+3897+eb84924d, kernel: 4.18.0-553.22.1.el8_10.x86_64, hostname: com10 LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \ HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/.config \ /usr/libexec/qemu-kvm \ -name guest=i-2-4127-VM,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-21617-i-2-4127-VM/master-key.aes"}' \ -machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram \ -accel kvm \ -cpu Cascadelake-Server-noTSX,mpx=off,hypervisor=on,pku=on,arch-capabilities=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on \ -m 2048 \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":2147483648}' \ -overcommit mem-lock=off \ -smp 2,sockets=2,dies=1,cores=1,threads=1 \ -uuid bf5f467f-c17a-4e6e-b3e2-1beb97941324 \ -smbios 'type=1,manufacturer=Apache Software Foundation,product=CloudStack KVM Hypervisor,serial=bf5f467f-c17a-4e6e-b3e2-1beb97941324,uuid=bf5f467f-c17a-4e6e-b3e2-1beb97941324' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=70,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \ -blockdev '{"driver":"host_device","filename":"/dev/storpool-byid/<volume-id>","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=c9e66d60139c4e90ab9a \ -device ide-cd,bus=ide.1,unit=0,id=ide0-1-0,bootindex=1 \ -netdev tap,fd=76,id=hostnet0,vhost=on,vhostfd=86 \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:02:03:0f:00:3c,bus=pci.0,addr=0x3 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=67,server=on,wait=off \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -audiodev '{"id":"audio1","driver":"none"}' \ -object '{"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/libvirt-vnc","endpoint":"server","verify-peer":true}' \ -vnc 10.xx.xx.60:0,password=on,tls-creds=vnc-tls-creds0,audiodev=audio1 \ -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \ -device i6300esb,id=watchdog0,bus=pci.0,addr=0x7 \ -watchdog-action none \ -incoming defer \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2026-04-21 08:20:25.095+0000: Domain id=21617 is tainted: high-privileges 2026-04-21 08:20:25.096+0000: Domain id=21617 is tainted: hook-script char device redirected to /dev/pts/25 (label charserial0) 2026-04-21T08:20:25.719385Z qemu-kvm: -device cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead 2026-04-21 08:20:56.634+0000: initiating migration 2026-04-21 08:20:57.921+0000: shutting down, reason=migrated 2026-04-21T08:20:57.922592Z qemu-kvm: terminating on signal 15 from pid 3930964 (/usr/sbin/libvirtd)

**log libvirt (dest host)
2026-04-21 08:20:54.470+0000: starting up libvirt version: 8.0.0, package: 23.2.module_el8.10.0+3869+b8959270 (AlmaLinux Packaging Team <packager@almalinux.org>, 2024-07-10-04:57:33, ), qemu version: 6.2.0qemu-kvm-6.2.0-53.module_el8.10.0+3897+eb84924d, kernel: 4.18.0-553.22.1.el8_10.x86_64, hostname: com11 LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \ HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/.config \ /usr/libexec/qemu-kvm \ -name guest=i-2-4127-VM,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-321-i-2-4127-VM/master-key.aes"}' \ -machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram \ -accel kvm \ -cpu Cascadelake-Server-noTSX,mpx=off,hypervisor=on,pku=on,arch-capabilities=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on \ -m 2048 \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":2147483648}' \ -overcommit mem-lock=off \ -smp 2,sockets=2,dies=1,cores=1,threads=1 \ -uuid bf5f467f-c17a-4e6e-b3e2-1beb97941324 \ -smbios 'type=1,manufacturer=Apache Software Foundation,product=CloudStack KVM Hypervisor,serial=bf5f467f-c17a-4e6e-b3e2-1beb97941324,uuid=bf5f467f-c17a-4e6e-b3e2-1beb97941324' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=42,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \ -blockdev '{"driver":"host_device","filename":"/dev/storpool-byid/<volume-id>","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=c9e66d60139c4e90ab9a \ -device ide-cd,bus=ide.1,unit=0,id=ide0-1-0,bootindex=1 \ -netdev tap,fd=46,id=hostnet0,vhost=on,vhostfd=55 \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:02:03:0f:00:3c,bus=pci.0,addr=0x3 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=41,server=on,wait=off \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -audiodev '{"id":"audio1","driver":"none"}' \ -object '{"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/libvirt-vnc","endpoint":"server","verify-peer":true}' \ -vnc 10.xx.xx.61:1,password=on,tls-creds=vnc-tls-creds0,audiodev=audio1 \ -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \ -device i6300esb,id=watchdog0,bus=pci.0,addr=0x7 \ -watchdog-action none \ -incoming defer \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2026-04-21 08:20:54.470+0000: Domain id=321 is tainted: high-privileges 2026-04-21 08:20:54.470+0000: Domain id=321 is tainted: hook-script char device redirected to /dev/pts/3 (label charserial0) 2026-04-21T08:20:55.588925Z qemu-kvm: -device cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead 2026-04-21T08:21:32.547875Z qemu-kvm: terminating on signal 15 from pid 1223112 (/usr/sbin/libvirtd) 2026-04-21 08:21:32.806+0000: shutting down, reason=shutdown

• After the failed migration:
  • The VM is not running on the source host
  • The VM is also not running on the destination host

Additional Observation

In some cases, the same error:
"Domain not found: no domain with matching name"
does not always result in a failed migration.

There are scenarios where:

• The VM is successfully migrated at the KVM/libvirt level
• But the CloudStack database is not updated to the new host

As a result:

• CloudStack still considers the VM on the source host
• When starting the VM, it may run again on the source host while already running on the destination host

This leads to:

• Duplicate VM running on two hosts
• Potential split-brain condition

versions

CloudStack version: 4.22.0.0
Hypervisor: KVM (libvirt 8.0.0, qemu-kvm 6.2.0)
OS: AlmaLinux / RHEL-based

The steps to reproduce the bug

1. Perform VM migration sequentially across multiple hosts (e.g., com01 → com02 → com03 → … → com24)
2. Repeat the migration process multiple times across the cluster until the issue occurs
3. this problem occurs when migrating from com10 to com11

What to do about it?

1. Ensure VM domain existence is properly validated on the source host before executing migration
2. Improve migration handling when the domain is missing:
   • Fail early with a clear and explicit error
   • Avoid proceeding with migration when the domain cannot be found
3. Improve logging and error reporting:
   • Provide detailed failure reason from agent/libvirt to management server
   • Ensure logs are consistent and not misleading