RT3754: check for NULL pointer

Fix both the caller to error out on malloc failure, as well as the
eventual callee to handle a NULL gracefully.

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>