Introduction
Vault supports a multi-server mode for high availability. This mode protects against outages by running multiple Vault servers. High availability mode is automatically enabled when using a data store that supports it.
Problem
Vault communicates with the other nodes using the IP addresses, that means the TLS certificates you provide to Vault, they need to have the DNS name as a certificate, but they also need a SAN (Subject Alternative Name) of the IP address in order for Vault to not complain about the certificate, cause Vault goes and communicates with the other nodes using the IP address, and if the IP address is not part of the certificate, you are going to get an error (bad certificate).
Vault servers advertise their API address to other Vault cluster members via the `api_addr` parameter in its runtime configuration. If you have TLS enabled on your listener for that address, other Vault cluster members will need to be able to complete a TLS handshake, which involves verifying the network address shown in the Subject of that certificate along with the certificate issuer from its local CA trust. If this is an IP address, it should be reflected in the SubjectAltName (SAN) of the certificate.
Note: In general, it’s considered best practice to advertise your Vault server’s `api_addr` with a DNS record reflected in your certificate, rather than an IP address.
Check this document for more information about Vault HA parameters.
Example Error
1 error occurred: * error unwrapping secondary token: Post "https://IP_ADDRESS.115:8200/v1/sys/wrapping/unwrap": x509: cannot validate certificate for IP_ADDRESS because it doesn't contain any IP SANs
Overview of possible solutions (if applicable)
Solution 1 - Using Vault PKI
Vault's PKI secrets engine can dynamically generate X.509 certificates on demand. You can do this following the below guide:
https://learn.hashicorp.com/tutorials/vault/pki-engine
For adding the SANs in your certificate, There are two parts to it.
- you have to have a PKI role that allows the URI sans, use the allowed_uri_sans property when you create the "server" role - you'll need to allow each of the alternate names.
- when you use vault to issue the cert, supply a uri_sans argument.
The vault command would look something like:
$ vault write pki/issue/server common_name="foobar.mydomain.com" ttl=2h uri_sans="foobar,barfoo"
Check this document for more information about Vault PKI sign certificate parameters.
Solution 2 - Using OpenSSL
The following is an example OpenSSL configuration to generate a self-signed certificate with the Subject Alternative Name (SAN) extension. This is provided as an illustrative example, and is not intended for production usage.
Subject Alternative Name extension is an extension of the X.509 specification described in RFC 5280, section 4.2.1.6 as follows:
The subject alternative name extension allows identities to be bound to the subject of the certificate. These identities may be included in addition to or in place of the identity in the subject field of the certificate. Defined options include an Internet electronic mail address, a DNS name, an IP address, and a Uniform Resource Identifier (URI).
To generate a self-signed certificate with the SAN extension using OpenSSL, we need to create a config first. Here’s what it can look like:
[req]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
countryName = XX
stateOrProvinceName = N/A
localityName = N/A
organizationName = Self-signed certificate
commonName = 120.0.0.1: Self-signed certificate
[req_ext]
subjectAltName = @alt_names
[v3_req]
subjectAltName = @alt_names
[alt_names]
IP.1 = 127.0.0.1
An important part here is the last one, where the IP address is set. It’s also possible to add additional IP addresses and hostnames in this section.
Save this config as san.cnf and pass it to OpenSSL:
$ openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyout key.pem -out cert.pem -config san.cnf
This will create a certificate with a private key. Let’s inspect it:
$ openssl x509 -in cert.pem -text -noout
The output should contain the IP address from the config:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 12776076004935403308 (0xb14dbd6f9991072c)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=XX, ST=N/A, L=N/A, O=Self-signed certificate, CN=127.0.0.1: Self-signed certificate
Validity ...
Subject: C=XX, ST=N/A, L=N/A, O=Self-signed certificate, CN=127.0.0.1: Self-signed certificate
Subject Public Key Info: ...
X509v3 extensions:
X509v3 Subject Alternative Name:
IP Address:127.0.0.1
Signature Algorithm: sha256WithRSAEncryption ...