Consul CA Certificate Rotation with No Downtime

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

CA Certificate rotation is a crucial process for maintaining secure communications in production environments. It involves replacing expiring or expired SSL/TLS certificates with new ones while ensuring uninterrupted service. The objective is to seamlessly update certificates without causing disruptions or errors for end-users.

Instructions

Rotate certificates when the Certificate Authority (CA) and certificates are due to expire

1. Create a new CA and certificate(s) on any one server.

2. Introduce the new CA and Combine the CA files.

   • Run the following command to combine the CA files

   cat <new-cert-dir>/consul-agent-ca.pem >> <existing-cert-dir>/consul-agent-ca.pem

3. Use openssl to check the combined CAs
   • Command

   openssl crl2pkcs7 -nocrl -certfile <existing-cert-dir>/consul-agent-ca.pem | openssl pkcs7 -print_certs -text -noout

   • Result example

     Certificate:
         Data:
             Version: 3 (0x2)
             Serial Number:
                 c8:33:f2:a2:4a:3f:8f:b9:34:e1:94:ae:a1:e1:4d:27 <---------- ca-certificate-1
             Signature Algorithm: ecdsa-with-SHA256
             Issuer: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp Inc., CN=Consul Agent CA 266115327497833593573162203714456472871
             Validity
                 Not Before: Mar  8 09:15:22 2023 GMT
                 Not After : Mar  9 09:15:22 2023 GMT <---------- ca-certificate-1
             Subject: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp 
     
     -------------------------
     Certificate:
         Data:
             Version: 3 (0x2)
             Serial Number:
                 24:36:e8:02:97:d4:cd:8e:47:a1:94:71:57:08:37:9d <----------- ca-certificate-2
             Signature Algorithm: ecdsa-with-SHA256
             Issuer: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp Inc., CN=Consul Agent CA 48137297603091723799054355290217002909
             Validity
                 Not Before: Mar  8 23:30:33 2023 GMT
                 Not After : Mar  6 23:30:33 2028 GMT <----------- ca-certificate-2
             Subject: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp

4. Copy the combined CA to all the servers and clients. Copy the CA to <existing-cert-dir>/consul-agent-ca.pem

5. While monitoring the logs, perform a rolling restart of the servers and then the client.

6. Replace the existing certs in <existing-cert-dir>with the new certs signed by the new CA.

7. Run rolling restart to load the new CA and certificate(s), while monitoring the logs.

8. Remove the old CA from the <existing-cert-dir>/consul-agent-ca.pem

9. Perform a rolling restart of the servers first then the client, while monitoring the logs.

Common TLS certificate errors

[ERROR] agent.server.rpc: failed to read byte: conn=from=100.65.29.108:56606
error="tls: failed to verify client certificate: x509: certificate has expired

Err: connection error: desc = "transport: Error while dialing x509: certificate has expired or is not yet valid:
current time 2023-03-08T10:13:33+11:00 is after 2023-03-07T01:29:09Z"

[ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.3:45021
error="remote error: tls: bad certificate"

[WARN] agent: certificate will expire soon:
metric=agent.tls.cert.expiry time_to_expiry=23h53m10.523503559s "

References :

• Secure Consul Agent Communication with TLS Encryption
• Update Consul Agents to Securely Communicate with TLS
• Secure Consul Agent Communication with Encryption and Certificates