OpenSSL: Fix the RSA_SSLV23_PADDING padding type
This also fixes the public function RSA_padding_check_SSLv23.
Commit 6555a89 changed the padding check logic in RSA_padding_check_SSLv23
so that padding is rejected if the nul delimiter byte is not immediately
preceded by at least 8 bytes containing 0x03. Prior to that commit the
padding is rejected if it *is* preceded by at least 8 bytes containing 0x03.
Presumably this change was made to be consistent with what it says in
appendix E.3 of RFC 5246. Unfortunately that RFC is in error, and the
original behaviour was correct. This is fixed in later errata issued for
that RFC.
Applications that use SSLv2 or call RSA_paddin_check_SSLv23 directly, or
use the RSA_SSLV23_PADDING mode may be impacted. The effect of the original
error is that an RSA message encrypted by an SSLv2 only client will fail to
be decrypted properly by a TLS capable server, or a message encrypted by a
TLS capable client will fail to decrypt on an SSLv2 only server. Most
significantly an RSA message encrypted by a TLS capable client will be
successfully decrypted by a TLS capable server. This last case should fail
due to a rollback being detected.
Thanks to D. Katz and Joel Luellwitz (both from Trustwave) for reporting
this issue.
CVE-2021-23839
https://github.com/openssl/openssl/commit/30919ab80a478f2d81f2e9acdcca3fa4740cd547