openssl req -utf8 mis-codes UTF8 strings in extension request

OpenSSL 3.0.14 and 3.3.3

Trying to create a CSR with a configuration file that has a Subject Alternative Name extension request with UTF8 special characters. This problem also affects arbitrary extensions that have special characters coded with ASN1:UTF8String in the configuration file.

> cat foo.cnf 
[req]
distinguished_name = req_distinguished_name
prompt             = no
utf8               = yes

[req_distinguished_name]
C  = DE
O  = Foobar
CN = Shortened name äöüß that would otherwise be too long

[san]
subjectAltName = otherName:commonName;UTF8:Long name äöüß with special chars that is too long for the SubjectDN

(Background is a bespoke requirement, that SubjectDN attributes which exceed RFC 5280 limits must be shortened and their full-length string copied into SAN.)

The config file is UTF8-encoded. Generate a key and the request:

> openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:brainpoolP256r1 -pkeyopt ec_param_enc:named_curve -out foo_key.pem
> openssl req -new -utf8 -config foo.cnf -reqexts san -key foo_key.pem -out foo_req.pem

The special characters in the SubjectDN are encoded correctly, whereas those in the SAN are not:

> openssl version
OpenSSL 3.0.14 4 Jun 2024 (Library: OpenSSL 3.0.14 4 Jun 2024)
> openssl req -text -noout -in foo_req.pem 
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: C = DE, O = Foobar, CN = Shortened name \C3\A4\C3\B6\C3\BC\C3\9F that would otherwise be too long
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:77:dd:1c:ae:65:3b:ea:ae:76:dd:84:e9:9c:c4:
                    d0:e7:55:6c:4c:d2:df:c1:44:14:13:e6:4b:43:c0:
                    de:1d:8f:5e:73:5e:a0:5f:61:52:5c:a5:07:ee:94:
                    92:c5:df:8f:e1:32:c2:fe:6d:dc:5f:1d:48:9f:a0:
                    32:d7:7e:dd:8c
                ASN1 OID: brainpoolP256r1
        Attributes:
            Requested Extensions:
                X509v3 Subject Alternative Name: 
                    othername: commonName::Long name äöüß with special chars that is too long for the SubjectDN
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:44:02:20:11:39:20:fc:2e:f4:34:40:af:d1:6c:a9:06:d7:
        e5:56:d2:b5:2d:ea:34:6a:18:db:cb:b3:99:9e:ff:ad:73:83:
        02:20:74:1e:b1:f9:c1:a2:8e:20:f6:f8:37:4f:9f:2c:25:f8:
        a9:76:ae:73:f2:1d:dd:3f:ae:ae:3b:7b:3f:54:ca:a9

OpenSSL 3.3.3 seems to have -nameopt utf8 as default, which is nice. But the SAN is just as incorrect:

> openssl version
OpenSSL 3.3.3 11 Feb 2025 (Library: OpenSSL 3.3.3 11 Feb 2025)
> openssl req -text -noout -in foo_req.pem
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: C=DE, O=Foobar, CN=Shortened name äöüß that would otherwise be too long
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:77:dd:1c:ae:65:3b:ea:ae:76:dd:84:e9:9c:c4:
                    d0:e7:55:6c:4c:d2:df:c1:44:14:13:e6:4b:43:c0:
                    de:1d:8f:5e:73:5e:a0:5f:61:52:5c:a5:07:ee:94:
                    92:c5:df:8f:e1:32:c2:fe:6d:dc:5f:1d:48:9f:a0:
                    32:d7:7e:dd:8c
                ASN1 OID: brainpoolP256r1
        Attributes:
            Requested Extensions:
                X509v3 Subject Alternative Name: 
                    othername: commonName::Long name äöüß with special chars that is too long for the SubjectDN
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:45:02:21:00:a6:4f:26:dc:bd:ca:7a:e5:f6:96:c4:05:e0:
        03:95:b9:a1:54:b5:cb:3c:ec:6d:5c:d5:f7:d2:7e:51:10:15:
        5c:02:20:2c:49:cd:42:22:b8:94:c7:65:a9:a3:c5:92:62:a8:
        fd:35:8a:b4:57:60:72:eb:f2:2d:79:c0:93:28:b1:9b:80

A hexdump of the name suggests that it has been double-encoded, i.e. each byte of the two-byte special characters has itself been encoded as two bytes.

00000000: 4c6f 6e67 206e 616d 6520 c383 c2a4 c383  Long name ......
00000010: c2b6 c383 c2bc c383 c29f 2077 6974 6820  .......... with 
00000020: 7370 6563 6961 6c20 6368 6172 7320 7468  special chars th
00000030: 6174 2069 7320 746f 6f20 6c6f 6e67 2066  at is too long f
00000040: 6f72 2074 6865 2053 7562 6a65 6374 444e  or the SubjectDN