cms: CHOICE types in `SET OF` are ordered as `SEQUENCE OF INTEGER`, not by their bytes

Open Beginner friendly
#2,538 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@brother-z is already working on this.

Since Nov 15, 2026.

  • #2539 by @brother-z — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
cryptography

Research direction

The issue names the four ValueOrd implementations in cert.rs, enveloped_data.rs, revocation.rs, and signed_data.rs. Read how each compares the DER encodings and check the ordering against the byte-string comparison described in the issue. Done means the affected CMS sets sort by their encoded bytes and decoding then re-encoding preserves DER order.

Written by the indexing model from the issue text.

Description

Found it while playing with EST server implementation and parsing example certs from RFC 7030 Appendix A.1 for /cacerts response. I encountered it in cert.rs, but same thing happens in enveloped_data.rs, revocation.rs and signed_data.rs.

Tl;dr: The comparison starts from correctly comparing headers, but if they're the same (same length certs) it tries to compare values of CHOICE and there it treats the whole blob (header included) as SEQUENCE OF INTEGER, makes a new, wrong header that can lie about length because bytes >=0x80 will get long encoding and then sorting falls apart.

The ValueOrd impls of CertificateChoices, RevocationInfoChoice, RecipientInfo and SignerIdentifier encode both values and compare the results with der_cmp:

self.to_der()?.der_cmp(&other.to_der()?)

to_der() returns Vec<u8>, and Vec<u8> gets DerOrd from the Vec<T> SEQUENCE OF impl, with each u8 as an INTEGER. So this compares the two encodings as if they were SEQUENCE OF INTEGER values. X.690 11.6 orders SET OF elements by comparing their encodings as octet strings, which for complete DER TLVs is Vec<u8>::cmp.

The two disagree:

use core::cmp::Ordering;
use der::DerOrd;

let a = vec![0x00u8, 0x80];
let b = vec![0x01u8, 0x00];
assert_eq!(a.cmp(&b), Ordering::Less);
assert_eq!(a.der_cmp(&b).unwrap(), Ordering::Less); // fails: Greater

As SEQUENCE OF INTEGER, a becomes 30 07 02 01 00 02 02 00 80 and b becomes 30 06 02 01 01 02 01 00. der_cmp compares these new headers first and gives wrong order.

SetOfVec sorts with this order when it is built and when it is decoded. So cms encodes CertificateSet, RevocationInfoChoices, RecipientInfos and SignerInfos (SignerInfo's derived ordering compares sid) in an order that is not DER, and decoding a correctly ordered set and encoding it again can change its bytes.

Example: the /cacerts response in RFC 7030 Appendix A.1 has four certificates. Decoded with cms, SignedData.certificates comes out as

NwN, OwO, OwN, NwO

The byte order is

OwO, NwN, NwO, OwN

(The RFC's own encoding uses neither order, so it isn't DER either.)

OwO and NwN are both 767 bytes, and NwO and OwN are both 775, yet each pair comes out swapped: their SEQUENCE OF INTEGER lengths differ (OwO 2599, NwN 2594, NwO 2640, OwN 2626).

Changing the four impls to Ok(self.to_der()?.cmp(&other.to_der()?)) fixes it. PR to follow.

Dominant language
Rust
Stars
347
Forks
200
Avg merge
15h 53m
Merged PRs (30d)
34

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from RustCrypto/formats

All issues in RustCrypto/formats

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.