RSA generation: Use more bits of 1/sqrt(2)

The old version always sets the top 2 bits, so the most significate byte
of the primes was always >= 0xC0. We now use 256 bits to represent
1/sqrt(2) = 0x0.B504F333F9DE64845...

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
GH: #10246

test/rsa_sp800_56b_test.c

diff --git a/test/rsa_sp800_56b_test.c b/test/rsa_sp800_56b_test.c
index 1e6ea8d..b928655 100644
--- a/test/rsa_sp800_56b_test.c
+++ b/test/rsa_sp800_56b_test.c
@@ -223,6 +223,8 @@
           && TEST_true(BN_set_word(p, 0x10))
           && TEST_false(rsa_check_prime_factor_range(p, 8, ctx))
           && TEST_true(BN_set_word(p, 0xB))
+          && TEST_false(rsa_check_prime_factor_range(p, 8, ctx))
+          && TEST_true(BN_set_word(p, 0xC))
           && TEST_true(rsa_check_prime_factor_range(p, 8, ctx))
           && TEST_true(BN_set_word(p, 0xF))
           && TEST_true(rsa_check_prime_factor_range(p, 8, ctx))