OpenSSL 3.5.0: timestamp reply generation fails with ML-DSA and SLH-DSA

Version: OpenSSL 3.5.1-dev (Library: OpenSSL 3.5.1-dev ) Freshly installed locally on an Ubuntu 24.10 machine.

The openssl-ts CLI utility fails to generate a timestamp reply with a ML-DSA or SLH-DSA keypair. This problem is very likely due to the limitations of the internal PKCS7 implementation, which already has trouble with providers (see this related issue from oqs-provider: https://github.com/open-quantum-safe/oqs-provider/issues/251).

The setup used to reproduce the issue is taken from https://www.jimby.name/techbits/recent/openssl_tsa/, replacing any mention of rsa:2048 with ML-DSA or SLH-DSA (for any parameter set).

All steps of the setup work nicely with rsa:2048, but the timestamp reply generation fails with ML-DSA/SLH-DSA:

openssl ts -reply -config ../rootca.conf -queryfile request.tsq -out response.tsr -provider default
Using configuration from ../rootca.conf
Enter PEM pass phrase:
Response has been generated.
800B9684617F0000:error:0700006C:configuration file routines:NCONF_get_string:no value:crypto/conf/conf_lib.c:316:group=tsa_config1 name=crypto_device
800B9684617F0000:error:0700006C:configuration file routines:NCONF_get_string:no value:crypto/conf/conf_lib.c:316:group=tsa_config1 name=other_policies
800B9684617F0000:error:10800094:PKCS7 routines:PKCS7_SIGNER_INFO_set:signing not supported for this key type:crypto/pkcs7/pk7_lib.c:392:
800B9684617F0000:error:17800076:time stamp routines:ts_RESP_sign:pkcs7 add signature error:crypto/ts/ts_rsp_sign.c:725:
800B9684617F0000:error:17800079:time stamp routines:TS_RESP_create_response:response setup error:crypto/ts/ts_rsp_sign.c:408:

Examining the timestamp reply produced shows the following:

openssl ts -reply -config ../rootca.conf -in response.tsr -text

Using configuration from ../rootca.conf
Status info:
Status: Rejected.
Status description: Error during signature generation.
Failure info: unspecified

TST info:
Not included.

With the upcoming widespread adoption of PQC in PKIs, timestamps will ultimately also have to be signed with PQC keypairs. It is also to be expected that hybridation will bring even more variety to the list of available signature algorithms, thus reinforcing the need for cryptoagility in tools such as openssl-ts.

The possibilities to fix this bug include upgrading the PKCS7 implementation for better support (potentially including providers too), or switching to the CMS implementation instead.

From the oqs-provider related issue, there has been previous designing work around ensuring provider compatibility for PCKS7 and CMS: https://github.com/openssl/openssl/blob/master/doc/designs/passing-algorithmidentifier-parameters.md