Treat all failures from EVP_DigestVerify() as a bad signature
Prior to 72ceb6a we treated all failures from the call to
EVP_DigestVerifyFinal() as if it were a bad signature, and failures in
EVP_DigestUpdate() as an internal error. After that commit we replaced
this with the one-shot function EVP_DigestVerify() and treated a 0 return
as a bad signature and a negative return as an internal error. However,
some signature errors can be negative (e.g. according to the docs if the
form of the signature is wrong). Therefore we should treat all <=0
returns as a bad signature.
This fixes a boringssl test failure.
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3756)
diff --git a/ssl/statem/statem_clnt.c b/ssl/statem/statem_clnt.c
index 711680e..7ab30bd 100644
--- a/ssl/statem/statem_clnt.c
+++ b/ssl/statem/statem_clnt.c
@@ -2272,11 +2272,7 @@
rv = EVP_DigestVerify(md_ctx, PACKET_data(&signature),
PACKET_remaining(&signature), tbs, tbslen);
OPENSSL_free(tbs);
- if (rv < 0) {
- al = SSL_AD_INTERNAL_ERROR;
- SSLerr(SSL_F_TLS_PROCESS_KEY_EXCHANGE, ERR_R_EVP_LIB);
- goto err;
- } else if (rv == 0) {
+ if (rv <= 0) {
al = SSL_AD_DECRYPT_ERROR;
SSLerr(SSL_F_TLS_PROCESS_KEY_EXCHANGE, SSL_R_BAD_SIGNATURE);
goto err;
