Consul CA Certificate Rotation with No Downtime – HashiCorp Help Center

The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

This article outlines the process to rotate Consul CA certificates without incurring downtime. Proper certificate management is essential for maintaining secure communication between Consul agents.

Problem

Rotating CA certificates in Consul can disrupt agent communication if not handled properly. This may lead to service downtime or loss of connectivity between Consul clients and servers.

Cause

CA certificate rotation issues can manifest through specific TLS-related errors during the process. Below are common causes, their corresponding log errors, and how they may be encountered

Cause

Error from Log

Expired

[ERROR] agent.server.rpc: failed to read byte: conn=from=100.65.29.108:56606
error="tls: failed to verify client certificate: x509: certificate has expired

Err: connection error: desc = "transport: Error while dialing x509: 
certificate has expired or is not yet valid:
current time 2023-03-08T10:13:33+11:00 is after 2023-03-07T01:29:09Z"

[ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.3:45021
error="remote error: tls: bad certificate"

Expiring soon

[WARN] agent: certificate will expire soon:
metric=agent.tls.cert.expiry time_to_expiry=23h53m10.523503559s "

Understanding these errors is crucial for diagnosing and resolving CA certificate rotation issues effectively. Always monitor logs during the rotation process to identify and address these errors promptly.

Solution

Note:

When instructed to restart the servers or run the consul reloadcommand, ALWAYS start with the followers and do the leader last
<new-cert-dir> - The path where the new CA and certificates exist
<existing-cert-dir> - The path where the current CA exists

Rotate certificates when the Certificate Authority (CA) and certificates are due to expire

Create a new CA and certificate(s) on any one server.
Introduce the new CA and Combine the CA files.
```
cat <new-cert-dir>/consul-agent-ca.pem >> <existing-cert-dir>/consul-agent-ca.pem
```

Use openssl to check the combined CAs

Command

openssl crl2pkcs7 -nocrl -certfile <existing-cert-dir>/consul-agent-ca.pem | openssl pkcs7 -print_certs -text -noout

Result example

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            c8:33:f2:a2:4a:3f:8f:b9:34:e1:94:ae:a1:e1:4d:27 <---------- ca-certificate-1
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp Inc., CN=Consul Agent CA 266115327497833593573162203714456472871
        Validity
            Not Before: Mar  8 09:15:22 2023 GMT
            Not After : Mar  9 09:15:22 2023 GMT <---------- ca-certificate-1
        Subject: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp 

-------------------------
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            24:36:e8:02:97:d4:cd:8e:47:a1:94:71:57:08:37:9d <----------- ca-certificate-2
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp Inc., CN=Consul Agent CA 48137297603091723799054355290217002909
        Validity
            Not Before: Mar  8 23:30:33 2023 GMT
            Not After : Mar  6 23:30:33 2028 GMT <----------- ca-certificate-2
        Subject: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp

Copy the combined CA to all the servers and clients. Copy the CA to <existing-cert-dir>/consul-agent-ca.pem
While monitoring the logs, perform a rolling restart of the servers and then the client.
Replace the existing certs in <existing-cert-dir>with the new certs signed by the new CA.
Run rolling restart to load the new CA and certificate(s), while monitoring the logs.
Remove the old CA from the <existing-cert-dir>/consul-agent-ca.pem
Perform a rolling restart of the servers first then the client, while monitoring the logs.

Additional Information :