Error tying to register ISO/Template with DirectDownload over HTTPS with Let's Encrypt

从 Agent 的证书处理和 /etc/cloudstack/agent/cloud.jks 入手，然后将其与 Java 的 /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts 以及位于 /etc/ssl/certs/ca-certificates.crt 的系统存储进行比较。使用 Let's Encrypt，针对 HTTPS URL 复现 ISO 或 Template 的直接下载，并考虑相关的 issue 和 PR；完成的标准是 Agent 无需针对每个 Zone 进行手动 workaround 即可验证证书。

problem

When using the "Direct Download" feature for an ISO or Template (bypassing Secondary Storage), the Agent fails to verify Let's Encrypt certificates due to the absence of Let's Encrypt’s main CA certificate.

Note: Let's Encrypt is widely used on the internet (>50% of all certificates).

ACS currently loads and trusts certificates exclusively from /etc/cloudstack/agent/cloud.jks and does not fall back to Java (/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts) or the system store (/etc/ssl/certs/ca-certificates.crt). Both of these contain the missing certificate (ISRG Root X1), which has been in use since 2015.

See: https://letsencrypt.org/certificates/

ISRG Root X1 is the current root of the trust chain (valid until 2030), after which it will be replaced by ISRG Root X2.

Recommendation: Add a fallback to Java’s trust store to avoid maintaining an ever-changing list of certificates.

Alternative: As a short-term fix, include the missing CA certificate (https://letsencrypt.org/certs/isrgrootx1.pem) in /etc/cloudstack/agent/cloud.jks for the next release, while a more sustainable solution is developed.

versions

We are running ACS 4.20.2 on Ubuntu 24.04. However, this issue likely affects all versions starting from 4.19, when the feature to bypass Secondary Storage was introduced.

Related issues and PRs:

• https://github.com/apache/cloudstack/issues/7929
• https://github.com/apache/cloudstack/pull/7693/changes
• https://github.com/apache/cloudstack/pull/7923/changes
• https://github.com/apache/cloudstack/pull/7932/changes
• https://github.com/apache/cloudstack/pull/11113/changes

The steps to reproduce the bug

1. When registering an ISO or Template for Direct Download, use any HTTPS URL whose TLS certificate is issued by Let's Encrypt.

What to do about it?

As a workaround, the following command can be run for each Zone to add the missing certificate. Note that this introduces additional manual steps for platform maintenance:

cmk upload templatedirectdownloadcertificate hypervisor="KVM" name="isrg-root-x1-2" certific
ate="$(curl -s https://letsencrypt.org/certs/isrgrootx1.pem)" zoneid="00000000-0000-0000-00000-000000000000"