An incompatibility has always existed between the format used for RSA
signatures and MDC2 using EVP or RSA_sign. This has become more apparent
when the dgst utility in OpenSSL 1.0.0 and later switched to using the
EVP_DigestSign functions which call RSA_sign.
This means that the signature format OpenSSL 1.0.0 and later used with
dgst -sign and MDC2 is incompatible with previous versions.
Add detection in RSA_verify so either format works.
Note: MDC2 is disabled by default in OpenSSL and very rarely used in practice.
diff --git a/CHANGES b/CHANGES
index 3712444..d4163ca 100644
--- a/CHANGES
+++ b/CHANGES
@@ -267,6 +267,13 @@
Changes between 1.0.0f and 1.0.1 [xx XXX xxxx]
+ *) The format used for MDC2 RSA signatures is inconsistent between EVP
+ and the RSA_sign/RSA_verify functions. This was made more apparent when
+ OpenSSL used RSA_sign/RSA_verify for some RSA signatures in particular
+ those which went through EVP_PKEY_METHOD in 1.0.0 and later. Detect
+ the correct format in RSA_verify so both forms transparently work.
+ [Steve Henson]
+
*) Some servers which support TLS 1.0 can choke if we initially indicate
support for TLS 1.2 and later renegotiate using TLS 1.0 in the RSA
encrypted premaster secret. As a workaround use the maximum pemitted