Fix pkeyutl/rsautl empty encrypt-input/decrypt-output handling

Also fix option processing in pkeyutl to allow use of (formerly)
"out-of-order" switches that were needless implementation limitations.

Handle documented "ENGINE" form with -keyform and -peerform.

Better handling of OPENSSL_NO_ENGINE and OPENSSL_NO_RSA.

RT2018

Reviewed-by: Rich Salz <rsalz@openssl.org>
diff --git a/doc/apps/pkeyutl.pod b/doc/apps/pkeyutl.pod
index 3c29b3a..d44f73a 100644
--- a/doc/apps/pkeyutl.pod
+++ b/doc/apps/pkeyutl.pod
@@ -11,10 +11,10 @@
 [B<-out file>]
 [B<-sigfile file>]
 [B<-inkey file>]
-[B<-keyform PEM|DER>]
+[B<-keyform PEM|DER|ENGINE>]
 [B<-passin arg>]
 [B<-peerkey file>]
-[B<-peerform PEM|DER>]
+[B<-peerform PEM|DER|ENGINE>]
 [B<-pubin>]
 [B<-certin>]
 [B<-rev>]
@@ -52,7 +52,7 @@
 
 the input key file, by default it should be a private key.
 
-=item B<-keyform PEM|DER>
+=item B<-keyform PEM|DER|ENGINE>
 
 the key format PEM, DER or ENGINE.
 
@@ -66,7 +66,7 @@
 
 the peer key file, used by key derivation (agreement) operations.
 
-=item B<-peerform PEM|DER>
+=item B<-peerform PEM|DER|ENGINE>
 
 the peer key format PEM, DER or ENGINE.
 
diff --git a/doc/apps/rsautl.pod b/doc/apps/rsautl.pod
index 6b98b51..92b8150 100644
--- a/doc/apps/rsautl.pod
+++ b/doc/apps/rsautl.pod
@@ -10,6 +10,7 @@
 [B<-in file>]
 [B<-out file>]
 [B<-inkey file>]
+[B<-keyform PEM|DER|ENGINE>]
 [B<-pubin>]
 [B<-certin>]
 [B<-sign>]
@@ -45,6 +46,10 @@
 
 the input key file, by default it should be an RSA private key.
 
+=item B<-keyform PEM|DER|ENGINE>
+
+the key format PEM, DER or ENGINE.
+
 =item B<-pubin>
 
 the input file is an RSA public key.