Recently I struggled with applying certificates to an Identity Manager 3.3 appliance. I ran into a few issues.
The customer supplied me with a PFX file. This file contains the certificate chain and the private key.
I extracted the certificate and the private key form the PFX file (see commands below on how to do this). When trying to apply the certificate chain and the private key to the Idm 3.3 appliance, I received an error: “Private key length is invalid”
After some googling I found this article: https://kb.vmware.com/s/article/56960
The KB states that 4096 bit certificates are not supported on vIDM 3.2 and higher due to FIPS regulations. vIDM 3.2 and later comes with FIPS and this cannot be disabled.
There are 2 workarounds:
- Use a 2048bit certificate
- Install vIDM 3.1, upgrade to 3.3 and apply the 4096bit certificate. Upgrading will not enable FIPS mode.
For me the only option was number 2.
I removed the 3.3 vIDM instance, deployed vIDM 3.1 and upgraded to 3.3. After this, I tried again to apply the 4096bit certificate.
Unfortunately I got another error: “The format of the private key is invalid”
To resolve this, follow the steps in this nice blogpost from @thepeb https://blogs.vmware.com/horizontech/2018/08/vmware-identity-manager-and-certificates.html
In short, the following commands should be executed using OpenSSL to generate the right certificate and private key pem files:
- Extract the certificate from the pfx file:
- openssl pkcs12 -in mycaservercert.pfx -nokeys -out mycaservercert.pem
- Extract the private key from the pfx file:
- openssl pkcs12 -in mycaservercert.pfx -nodes -nocerts -out mycaservercertkey.pem
- Convert from PKCS #8 to a PKCS #1 private key
- openssl rsa -in mycaservercertkey.pem -check -out mycaservercertkeyrsa.pem
Step 3 is the one that did the trick on vIDM 3.1. This version only supports PKCS #1 private keys. Hence the error I got, because my private key was still in PKCS #8. From version 3.2 on also PKCS #8 is supported.