Introduction
Problem
When VMware Vcenter is configured to use the Vault KMIP Secrets Engine as KMS service, VMware Vcenter is unable to reconnect to the KMS Service (the Vault KMIP Secrets Engine) after a restart of the Vault Instance, or after a Vault leadership re-election. At this point, manual intervention is required and the trust has to be re-established from the VMware Vcenter side, using the establish trust option.
Prerequisites (if applicable)
- Vault Enterprise with KMIP Secrets Engine Enabled
- VMware Vcenter
Cause
When the Vault Service is restarted or a Leadership election occurs, the listener certificate used by Vault KMIP Secrets Engine is automatically regenerated by the Vault KMIP Secrets Engine with a validity of 365 days, if VMware Vcenter is configured to trust the listener certificate (KMS leaf certificate), VMware Vcenter will no longer trust the certificate after it has been automatically regenerated by Vault and the trust has to be re-established from the VMware Vcenter side.
If Vault Performance Standby nodes are being used in the Vault Cluster used as KMS service by VMware Vcenter, the listener certificate used by Vault KMIP Secrets Engine by each Vault instance will contain a different listener Certificate as the Certificate is being generated at Vault Startup, and the issue might occur if a load balancer is used to distribute traffic to the Vault KMIP Secrets Engine between multiple Vault instances.
The VMware Vcenter documentation contains the following two options:
You can complete the trust setup, that is, make vCenter Server trust
the KMS, either by trusting the KMS or by uploading a KMS certificate.
You have two options:
Trust the certificate explicitly by using the Upload KMS certificate option.
Upload a KMS leaf certificate or the KMS CA certificate to vCenter Server
by using the Make vCenter Trust KMS option.
Note:
If you upload the root CA certificate or the intermediate CA certificate,
vCenter Server trusts all certificates that are signed by that CA.
For strong security, upload a leaf certificate or an intermediate
CA certificate that the KMS vendor controls.
However, uploading the vault-ca.pem obtained from the Vault KMIP Secrets Engine using the KMS CA certificate option within the VMware Vcenter does not change this behavior, which is unexpected.
Overview of possible solutions (if applicable)
Solutions:
The generated vault-ca.pem contains 2 CA certificates called:
- CN = vault-kmip-default-intermediate
- CN = vault-kmip-default
VMware Vcenter only imports the first certificate from the vault-ca.pem file, which is the CN = vault-kmip-default-intermediate certificate instead of the required CN = vault-kmip-default certificate.
The following command can be used to extract two separate pem files from the generated vault-ca.pem file:
awk '
split_after == 1 {n++;split_after=0}
/-----END CERTIFICATE-----/ {split_after=1}
{print > "ca_cert" n ".pem"}' vault-ca.pem
The expected outcome is that two pem files are generated, containing one certificate each:
openssl x509 -noout -text -in 'ca_cert.pem' | grep -i subject:
Subject: CN = vault-kmip-default-intermediate
openssl x509 -noout -text -in 'ca_cert1.pem' | grep -i subject:
Subject: CN = vault-kmip-default
Outcome
After specifying the ca_cert1.pem file, which only contains the vault-kmip-default Root CA certificate using the vCenter Trust KMS --> Upload KMS Certificate option, vCenter is able to reconnect to the KMS Service (The Vault KMIP Secrets Engine) after a restart of Vault or a Leadership election without any manual intervention.