The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.
All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.
Introduction
This article outlines the process to rotate Consul CA certificates without incurring downtime. Proper certificate management is essential for maintaining secure communication between Consul agents.
Problem
Rotating CA certificates in Consul can disrupt agent communication if not handled properly. This may lead to service downtime or loss of connectivity between Consul clients and servers.
Cause
CA certificate rotation issues can manifest through specific TLS-related errors during the process. Below are common causes, their corresponding log errors, and how they may be encountered
Cause | Error from Log |
Expired |
[ERROR] agent.server.rpc: failed to read byte: conn=from=100.65.29.108:56606
Err: connection error: desc = "transport: Error while dialing x509:
[ERROR] agent.server.rpc: failed to read byte: conn=from=127.0.0.3:45021
|
Expiring soon |
[WARN] agent: certificate will expire soon:
|
Understanding these errors is crucial for diagnosing and resolving CA certificate rotation issues effectively. Always monitor logs during the rotation process to identify and address these errors promptly.
Solution
- When instructed to restart the servers or run the
consul reload
command, ALWAYS start with the followers and do the leader last <new-cert-dir>
- The path where the new CA and certificates exist-
<existing-cert-dir>
- The path where the current CA exists
Rotate certificates when the Certificate Authority (CA) and certificates are due to expire
-
Create a new CA and certificate(s) on any one server.
-
Introduce the new CA and Combine the CA files.
-
Run the following command to combine the CA files
cat <new-cert-dir>/consul-agent-ca.pem >> <existing-cert-dir>/consul-agent-ca.pem
-
- Use
openssl
to check the combined CAs- Command
openssl crl2pkcs7 -nocrl -certfile <existing-cert-dir>/consul-agent-ca.pem | openssl pkcs7 -print_certs -text -noout
-
Result example
Certificate: Data: Version: 3 (0x2) Serial Number: c8:33:f2:a2:4a:3f:8f:b9:34:e1:94:ae:a1:e1:4d:27 <---------- ca-certificate-1 Signature Algorithm: ecdsa-with-SHA256 Issuer: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp Inc., CN=Consul Agent CA 266115327497833593573162203714456472871 Validity Not Before: Mar 8 09:15:22 2023 GMT Not After : Mar 9 09:15:22 2023 GMT <---------- ca-certificate-1 Subject: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp ------------------------- Certificate: Data: Version: 3 (0x2) Serial Number: 24:36:e8:02:97:d4:cd:8e:47:a1:94:71:57:08:37:9d <----------- ca-certificate-2 Signature Algorithm: ecdsa-with-SHA256 Issuer: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp Inc., CN=Consul Agent CA 48137297603091723799054355290217002909 Validity Not Before: Mar 8 23:30:33 2023 GMT Not After : Mar 6 23:30:33 2028 GMT <----------- ca-certificate-2 Subject: C=US, ST=CA, L=San Francisco/street=101 Second Street/postalCode=94105, O=HashiCorp
- Copy the combined CA to all the servers and clients. Copy the CA to
<existing-cert-dir>/consul-agent-ca.pem
-
While monitoring the logs, perform a rolling restart of the servers and then the client.
-
Replace the existing certs in
<existing-cert-dir>
with the new certs signed by the new CA. -
Run rolling restart to load the new CA and certificate(s), while monitoring the logs.
-
Remove the old CA from the
<existing-cert-dir>/consul-agent-ca.pem
-
Perform a rolling restart of the servers first then the client, while monitoring the logs.
Additional Information :
- Secure Consul Agent Communication with TLS Encryption
- Update Consul Agents to Securely Communicate with TLS
- Secure Consul Agent Communication with Encryption and Certificates