OpenSSL 3.3.0 with OQS Provider: Ml-DSA-87 Algorithms Not Supported Correctly

[laaurii00]

I am using Ubuntu 22.04 and have installed OpenSSL 3.3.0 along with the liboqs and oqs-provider. However, I have encountered an issue where, when establishing communication using ML-DSA-87 algorithms and RSA KEM method, OpenSSL does not support the established of oepnSSL communication with ML-DSA-87 digital signature algorithms correctly.

Environment:
Operating System: Ubuntu 22.04
OpenSSL Version: 3.3.0
OQS Provider Version: 0.8.1-dev
OQS Provider Features: liboqs and oqs-provider enabled (I have reinstall the last version of both libraries in my server and client machine).
OpenSSL Configuration:
[openssl_init]
providers = provider_sect
ssl_conf = ssl_sect

• List of providers to load

[provider_sect]
default = default_sect
oqsprovider = oqsprovider_sect

[default_sect]
activate = 1

[oqsprovider_sect]
activate = 1

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Groups = RSA:x25519

• OpenSSL providers:

  Providers:
    default
      name: OpenSSL Default Provider
      version: 3.3.0
      status: active
    oqsprovider
      name: OpenSSL OQS Provider
      version: 0.8.1-dev
      status: active

• OpenSSL List of KEM Algorithms:

  { 1.2.840.113549.1.1.1, 2.5.8.1.1, RSA, rsaEncryption } @ default
  { 1.2.840.10045.2.1, EC, id-ecPublicKey } @ default
  { 1.3.101.110, X25519 } @ default
  { 1.3.101.111, X448 } @ default
  frodo640aes @ oqsprovider
  p256_frodo640aes @ oqsprovider
  x25519_frodo640aes @ oqsprovider
  frodo640shake @ oqsprovider
  p256_frodo640shake @ oqsprovider
  x25519_frodo640shake @ oqsprovider
  frodo976aes @ oqsprovider
  p384_frodo976aes @ oqsprovider
  x448_frodo976aes @ oqsprovider
  frodo976shake @ oqsprovider
  p384_frodo976shake @ oqsprovider
  x448_frodo976shake @ oqsprovider
  frodo1344aes @ oqsprovider
  p521_frodo1344aes @ oqsprovider
  frodo1344shake @ oqsprovider
  p521_frodo1344shake @ oqsprovider
  mlkem512 @ oqsprovider
  p256_mlkem512 @ oqsprovider
  x25519_mlkem512 @ oqsprovider
  mlkem768 @ oqsprovider
  p384_mlkem768 @ oqsprovider
  x448_mlkem768 @ oqsprovider
  X25519MLKEM768 @ oqsprovider
  SecP256r1MLKEM768 @ oqsprovider
  mlkem1024 @ oqsprovider
  p521_mlkem1024 @ oqsprovider
  SecP384r1MLKEM1024 @ oqsprovider
  bikel1 @ oqsprovider
  p256_bikel1 @ oqsprovider
  x25519_bikel1 @ oqsprovider
  bikel3 @ oqsprovider
  p384_bikel3 @ oqsprovider
  x448_bikel3 @ oqsprovider
  bikel5 @ oqsprovider
  p521_bikel5 @ oqsprovider
  hqc128 @ oqsprovider
  p256_hqc128 @ oqsprovider
  x25519_hqc128 @ oqsprovider
  hqc192 @ oqsprovider
  p384_hqc192 @ oqsprovider
  x448_hqc192 @ oqsprovider
  hqc256 @ oqsprovider
  p521_hqc256 @ oqsprovider

• openssl list -signature-algorithms:

{ 1.2.840.113549.1.1.1, 2.5.8.1.1, RSA, rsaEncryption } @ default
  { 1.2.840.10040.4.1, 1.2.840.10040.4.3, 1.3.14.3.2.12, 1.3.14.3.2.13, 1.3.14.3.2.27, DSA, DSA-old, DSA-SHA, DSA-SHA1, DSA-SHA1-old, dsaEncryption, dsaEncryption-old, dsaWithSHA, dsaWithSHA1, dsaWithSHA1-old } @ default
  { 1.3.101.112, ED25519 } @ default
  { 1.3.101.113, ED448 } @ default
  { 1.2.156.10197.1.301, SM2 } @ default
  ECDSA @ default
  HMAC @ default
  SIPHASH @ default
  POLY1305 @ default
  CMAC @ default
  mldsa44 @ oqsprovider
  p256_mldsa44 @ oqsprovider
  rsa3072_mldsa44 @ oqsprovider
  mldsa44_pss2048 @ oqsprovider
  mldsa44_rsa2048 @ oqsprovider
  mldsa44_ed25519 @ oqsprovider
  mldsa44_p256 @ oqsprovider
  mldsa44_bp256 @ oqsprovider
  mldsa65 @ oqsprovider
  p384_mldsa65 @ oqsprovider
  mldsa65_pss3072 @ oqsprovider
  mldsa65_rsa3072 @ oqsprovider
  mldsa65_p256 @ oqsprovider
  mldsa65_bp256 @ oqsprovider
  mldsa65_ed25519 @ oqsprovider
  mldsa87 @ oqsprovider
  p521_mldsa87 @ oqsprovider
  mldsa87_p384 @ oqsprovider
  mldsa87_bp384 @ oqsprovider
  mldsa87_ed448 @ oqsprovider
  falcon512 @ oqsprovider
  p256_falcon512 @ oqsprovider
  rsa3072_falcon512 @ oqsprovider
  falconpadded512 @ oqsprovider
  p256_falconpadded512 @ oqsprovider
  rsa3072_falconpadded512 @ oqsprovider
  falcon1024 @ oqsprovider
  p521_falcon1024 @ oqsprovider
  falconpadded1024 @ oqsprovider
  p521_falconpadded1024 @ oqsprovider
  sphincssha2128fsimple @ oqsprovider
  p256_sphincssha2128fsimple @ oqsprovider
  rsa3072_sphincssha2128fsimple @ oqsprovider
  sphincssha2128ssimple @ oqsprovider
  p256_sphincssha2128ssimple @ oqsprovider
  rsa3072_sphincssha2128ssimple @ oqsprovider
  sphincssha2192fsimple @ oqsprovider
  p384_sphincssha2192fsimple @ oqsprovider
  sphincsshake128fsimple @ oqsprovider
  p256_sphincsshake128fsimple @ oqsprovider
  rsa3072_sphincsshake128fsimple @ oqsprovider
  mayo1 @ oqsprovider
  p256_mayo1 @ oqsprovider
  mayo2 @ oqsprovider
  p256_mayo2 @ oqsprovider
  mayo3 @ oqsprovider
  p384_mayo3 @ oqsprovider
  mayo5 @ oqsprovider
  p521_mayo5 @ oqsprovider
  CROSSrsdp128balanced @ oqsprovider

Issue Details:
Working with keys and certificates generated with ML-DSA-87:

openssl genpkey -algorithm mldsa87 -out ca.key
time openssl req -key ca.key -new -x509 -out ca.pem -subj "XXX"

time openssl genpkey -algorithm mldsa87 -out server.key
openssl req -key server.key -new -out server.csr -subj "XXX"
time openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out server.crt -days 365

Try to establish connection:

openssl s_client -connect XXX:4433 -cert client.crt -key client.key
openssl s_server -cert server.crt -key server.key

Server Response:

**Using default temp DH parameters
ACCEPT
ERROR
80AB72C24A7F0000:error:0A00041B:SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error:ssl/record/rec_layer_s3.c:907:SSL alert number 51
shutting down SSL
CONNECTION CLOSED**

Client Response:

---
No client certificate CA names sent
Peer signature type: mldsa87
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 12498 bytes and written 406 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Expected Behavior:
I expect the ML-DSA-87 or ML-DSA-44 (also tried) algorithms to be correctly recognized and supported by OpenSSL by the server, as is working with digital signatures as Falcon512, falcon1024, Mayo5, etc.

Any help or insights would be greatly appreciated!

[SWilson4]

Hi @laaurii00, thanks for the report. I'm going to move this issue to the oqs-provider repo, since it seems to be related to that project.

[SWilson4]

@baentsch We received this report in liboqs, and I didn't immediately notice a problem with the config—can you take a look?

[ashman-p]

Hi @laaurii00, For what its worth, I tried to repeat your steps but was able to make the connection.
was able to connect without using the client cert/key (because i did not create a pair and you did not enable mutual auth).

apps/openssl s_server -cert mldsa87_server.crt -key mldsa87_server.key -port 4444
apps/openssl s_client -port 4444 

apps/openssl s_server -sigalgs mldsa87 -cert mldsa87_server.crt -key mldsa87_server.key -port 4444
apps/openssl s_client -port 4444 

Perhaps you could post your test certs and keys. We can debug the tests.

[laaurii00]

I attach the server.crt, server.key, client.crt and client.key: Certs_Issue.zip

openssl s_client -connect XXX:4433 -cert client.crt -key client.key
openssl s_server -cert server.crt -key server.key

Thank you.