Unable to launch a vm with a template which is direct download template and having nfs metalink

Start with the KVMStorageProcessor and MetalinkDirectTemplateDownloader entry points shown in the agent.log trace, then reproduce the NFS metalink launch using the XML and steps in the issue. Trace the mount, copy, and cleanup results; done means a direct-download template with an NFS metalink launches a VM successfully without the reported mount-point and false-success errors.

problem

Unable to launch a vm with a template which is direct download template and having nfs metalink

versions

ACS 4.22.1

The steps to reproduce the bug

1. Create a metalink file

Example

<?xml version="1.0" encoding="UTF-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/">
  <files>
    <file name="ubuntu24-cloud-image-root-password-with-tools.qcow2">
      <size>10485760</size>
      <resources>
        <url type="nfs" preference="100">nfs://10.0.35.201/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2</url>
      </resources>
    </file>
  </files>
</metalink>

2. Expose it via http server

python3 -m http.server 8081 --bind 0.0.0.0

3. Register it a direct download template in cloudstack

4. Launch a vm

5. Exception in agent.log

2026-08-19 04:36:53,965 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Request:Seq 1-6884877930342650195:  { Cmd , MgmtId: 32985734710148, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.agent.directdownload.MetalinkDirectDownloadCommand":{"url":"http://10.0.35.201:8081/good-nfs.metalink","templateId":"202","destData":{"origUrl":"http://10.0.35.201:8081/good-nfs.metalink","uuid":"e50a9618-71f4-4b68-881a-54a2527d3e02","id":"202","format":"QCOW2","accountId":"2","hvm":"true","displayText":"test-nfs-metalink","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"24518d40-97cd-3433-bafa-dcb0291c03f9","name":"ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=24518d40-97cd-3433-bafa-dcb0291c03f9","isManaged":"false"}},"name":"202-2-7772abb0-fd86-32ff-add7-ef855cd70293","size":"(6580590.5000 TB) 7235435666033759572","hypervisorType":"KVM","bootable":"false","uniqueName":"202-2-7772abb0-fd86-32ff-add7-ef855cd70293","directDownload":"true","deployAsIs":"false","followRedirects":"false"},"destPool":{"uuid":"24518d40-97cd-3433-bafa-dcb0291c03f9","name":"ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=24518d40-97cd-3433-bafa-dcb0291c03f9","isManaged":"false"},"headers":{},"connectTimeout":"5000","soTimeout":"5000","templateSize":"(6580590.5000 TB) 7235435666033759572","format":"QCOW2","followRedirects":"false","wait":"10800","bypassHostMaintenance":"false"}}] }
2026-08-19 04:36:53,965 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Processing command: org.apache.cloudstack.agent.directdownload.MetalinkDirectDownloadCommand

2026-08-19 04:36:53,965 DEBUG [storage.resource.StorageSubsystemCommandHandlerBase] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Executing command MetalinkDirectDownloadCommand: [{"url":"http://10.0.35.201:8081/good-nfs.metalink","templateId":202,"destData":{"origUrl":"http://10.0.35.201:8081/good-nfs.metalink","uuid":"e50a9618-71f4-4b68-881a-54a2527d3e02","id":202,"format":"QCOW2","accountId":2,"hvm":true,"displayText":"test-nfs-metalink","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"24518d40-97cd-3433-bafa-dcb0291c03f9","name":"ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","id":1,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=24518d40-97cd-3433-bafa-dcb0291c03f9","isManaged":false}},"name":"202-2-7772abb0-fd86-32ff-add7-ef855cd70293","size":7235435666033759572,"hypervisorType":"KVM","bootable":false,"uniqueName":"202-2-7772abb0-fd86-32ff-add7-ef855cd70293","directDownload":true,"deployAsIs":false,"followRedirects":false},"destPool":{"uuid":"24518d40-97cd-3433-bafa-dcb0291c03f9","name":"ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","id":1,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=24518d40-97cd-3433-bafa-dcb0291c03f9","isManaged":false},"headers":{},"connectTimeout":5000,"soTimeout":5000,"templateSize":7235435666033759572,"format":"QCOW2","followRedirects":false,"wait":10800,"bypassHostMaintenance":false}].

2026-08-19 04:36:53,965 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Verifying temporary location for downloading the template exists on the host
2026-08-19 04:36:53,967 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Checking for free space on the host for downloading the template with physical size: 413 and virtual size: 7235435666033759572
2026-08-19 04:36:53,967 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Executing command [/bin/bash -c df --output=avail /var/lib/libvirt/images -B 1 | tail -1 ].
2026-08-19 04:36:53,973 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Successfully executed process [68470] for command [/bin/bash -c df --output=avail /var/lib/libvirt/images -B 1 | tail -1 ].
2026-08-19 04:36:53,973 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Trying to fetch storage pool 24518d40-97cd-3433-bafa-dcb0291c03f9 from libvirt
2026-08-19 04:36:53,974 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Looking for libvirtd connection at: qemu:///system
2026-08-19 04:36:53,978 DEBUG [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Successfully refreshed pool 24518d40-97cd-3433-bafa-dcb0291c03f9 Capacity: (3.4170 TB) 3757022642176 Used: (2.7297 TB) 3001299238912 Available: (703.82 GB) 755723403264
2026-08-19 04:36:53,987 ERROR [cloud.utils.UriUtils] (AgentRequest-Handler-3:[]) (logid:fe7c1979) no hash tag in XML response...
2026-08-19 04:36:53,987 INFO  [direct.download.MetalinkDirectTemplateDownloader] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Metalink downloader created, metalink URL: http://10.0.35.201:8081/good-nfs.metalink parsed - 1 urls and 0 checksums found
2026-08-19 04:36:53,987 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Trying to download template
2026-08-19 04:36:53,987 INFO  [direct.download.MetalinkDirectTemplateDownloader] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Trying to download Template from URL: nfs://10.0.35.201/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2
2026-08-19 04:36:53,987 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Executing command [/bin/bash -c mount -t nfs 10.0.35.201:/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ].
2026-08-19 04:36:53,999 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Execution of process [68473] for command [/bin/bash -c mount -t nfs 10.0.35.201:/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] failed.
2026-08-19 04:36:53,999 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Exit value of process [68473] for command [/bin/bash -c mount -t nfs 10.0.35.201:/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] is [32].
2026-08-19 04:36:53,999 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Process [68473] for command [/bin/bash -c mount -t nfs 10.0.35.201:/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] encountered the error: [mount.nfs: mount point /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 does not exist].
2026-08-19 04:36:53,999 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Executing command [/bin/bash -c cp /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/96fa2884-a398-48f9-acb6-7d00e894520f.qcow2 ].
2026-08-19 04:36:54,005 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Execution of process [68475] for command [/bin/bash -c cp /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/96fa2884-a398-48f9-acb6-7d00e894520f.qcow2 ] failed.
2026-08-19 04:36:54,006 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Exit value of process [68475] for command [/bin/bash -c cp /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/96fa2884-a398-48f9-acb6-7d00e894520f.qcow2 ] is [1].
2026-08-19 04:36:54,006 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Process [68475] for command [/bin/bash -c cp /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/96fa2884-a398-48f9-acb6-7d00e894520f.qcow2 ] encountered the error: [cp: cannot stat '/mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2': No such file or directory].
2026-08-19 04:36:54,006 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Executing command [/bin/bash -c umount /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ].
2026-08-19 04:36:54,011 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Execution of process [68476] for command [/bin/bash -c umount /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] failed.
2026-08-19 04:36:54,011 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Exit value of process [68476] for command [/bin/bash -c umount /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] is [32].
2026-08-19 04:36:54,011 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Process [68476] for command [/bin/bash -c umount /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] encountered the error: [umount: /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04: no mount point specified.].
2026-08-19 04:36:54,011 INFO  [direct.download.MetalinkDirectTemplateDownloader] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Successfully downloaded Template from URL: nfs://10.0.35.201/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2

What to do about it?

Should be able to successfully launch a vm from metalink url which has a nfs file

https://docs.cloudstack.apache.org/en/4.22.1.0/adminguide/templates/_bypass-secondary-storage-kvm.html