VM deployment randomly fails on XCP-ng 8.3 due to stale template_spool_ref

Start by tracing how template_spool_ref entries are created and handled when a template copy or base volume is deleted, using the reported XCP-ng deployment failure as the reproduction signal. Determine the appropriate cleanup behavior and verify that deploying from an affected template no longer attempts to access a missing VDI.

problem

We are experiencing intermittent/random VM deployment failures in CloudStack 4.22.1 with XCP-ng 8.3.

During the investigation, we observed that the template copy/base volume associated with the deployment is no longer present in the primary storage. However, the corresponding database entry still exists in the template_spool_ref table.

When CloudStack subsequently attempts to deploy a VM from the affected template, it appears to retrieve the stale storage reference from the database and attempts to access a VDI that no longer exists.

The deployment then fails with an XCP-ng API error indicating that the VDI UUID is invalid.

versions

Apache CloudStack: 4.22.1
Hypervisor: XCP-ng 8.3
Storage: Presetup [iSCSI]

The steps to reproduce the bug

VM Launch is failing below is observed in logs:

2026-09-08 15:02:51,295 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-51:[ctx-21e4a1d0, job-5595/job-5596, ctx-af6a6525]) (logid:0b1aca2f) Unable to create volume [{"name":"ROOT-449","uuid":"6871902e-df60-4d42-9b9a-7dd2851e0c1b"}] due to [com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: 85032e3f-3d1a-4336-af29-4b7039fc71a0 failed due to The uuid you supplied was invalid.].
2026-09-08 15:02:51,295 WARN  [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-51:[ctx-21e4a1d0, job-5595/job-5596, ctx-af6a6525]) (logid:0b1aca2f) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create volume [{"name":"ROOT-449","uuid":"6871902e-df60-4d42-9b9a-7dd2851e0c1b"}] due to [com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: 85032e3f-3d1a-4336-af29-4b7039fc71a0 failed due to The uuid you supplied was invalid.].

What to do about it?

Should the corresponding template_spool_ref entry be removed or marked appropriately when the template copy is deleted?