Description
The certified key selection implemented by rustls_client_hello_select_certified_key() currently does not take the limitations of the later defined rustls_connection into account. This leads to handshake failures.
Assuming you have a server with 2 certified keys: P-384 and RSA.
The rustls_server_config has:
- TLS 1.2 and 1.3 enabled
- All 1.3 ciphers and only the RSA 1.2 ciphers
A client connecting with TLS 1.2, the rustls_client_hello_select_certified_key() will chose the P-384 certified key (since the client has the necessary ciphers), create the rustls_connection, start the handshake and fail. The connection simply is unable to do RSA with the P-384 certified key.
The fix would be for rustls_client_hello_select_certified_key() to take the server ciphers into account as well. This is a bit awkward, since the certified key needs to be selected before the rustls_connection for the handshake is created.
At the moment, this is a rare scenario. But selecting the certified key on incomplete understanding of the TLS properties involved, could lead to more common failures in the future. When a new rustls version is released and disables ciphers or particular combinations of them.
Supporting multiple certified keys in a server was helpful in the introduction of EC. The continued support of such a feature via rustls_client_hello_select_certified_key() carries good intentions, however this case shows that it is incomplete.
I guess, either certified key selection is implemented deep inside the rustls handshake itself or it needs to be abandoned.
Opinions?
PS. minor detail: the error of such a failure gives [7107] peer is incompatible: reason omitted. Where does this reason omitted come from? In the rustls code I see only cases of this error with descriptions.