v3.0.13: cms decryption fails when EVP_PKEY_EC engine is loaded: crypto/evp/ctrl_params_translate.c:1211: OpenSSL internal error: Assertion failed: ctx->action_type != NONE

The problem was first found with openssl 3.0.13. I'm using a debug build on x86-64 system (reproducer was build and run on Ubuntu 20.04 with gcc 9.4).

I'm using an engine to support ECDSA operations and run into some issues when first trying to use CMS operations with that engine.

I was able to reproduce the issue with a EVP_PKEY_EC test engine that forwards all calls to the builtin openssl method (based on code by El Mostafa IDRASSI). This test engine and the reproducer can be found in https://github.com/myrkr/openssl-testengine as I had to modify the engine to avoid another problem when EVP_PKEY_FLAG_AUTOARGLEN is set).

The test setup assumes that the openssl version to test against is found in ~/git/openssl. I used the current openssl-3.0 branch for testing.

I first created a test CA and a test key (script for setup in found in the make-test-setup directory). The created test data were stored in the data subdirectory.

Then the script test-extract is run (building the engine when not already done).

Expected output:

~/git/openssl-testengine (main u=)$ ./test-extract
Extracting without engine
Extracting with forwarding engine

Actual output:

~/git/openssl-testengine (main u=)$ ./test-extract
Extracting without engine
Extracting with forwarding engine
crypto/evp/ctrl_params_translate.c:1211: OpenSSL internal error: Assertion failed: ctx->action_type != NONE

Just commenting the assertion in the error location fixes the problem for me.