Introduction
Validating SSL certificates require a strict file format with the appropriate ordering of intermediate certificates. Terraform Enterprise uses a list of PEM-formatted CA certificates to allow the containers to verify SSL certificates with private Certificate Authorities.
Problem
Typically these issues present with an x509: certificate signed by unknown authority error. This could be during the install process, in a container's logs, or in a Terraform plan or apply. The error is typically prefixed by the URL of the failed request. This error means that one or more of the CAs that signed the certificates served by that URL are unable to be verified.
Procedure
First, ensure that the Terraform Enterprise's CA certificate settings contain the CA certificates in the correct order and without any extra information, such as Bag Attributes: https://support.hashicorp.com/hc/en-us/articles/4406543390099-Terraform-Enterprise-Order-of-certificates-in-SSL-cert-files
The openssl and curl commands can help dig deeper into this issue. This article will use google.com as an example. For this example, imagine an error occurred in validating the https://google.com SSL certificate, stating x509: certificate signed by unknown authority.
First we should verify what certificates the site is serving. This can be done with openssl s_client -showcerts -connect google.com:443 </dev/null. Here's a trimmed output for illustration purposes:
CONNECTED(00000198) depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1 verify return:1 depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3 verify return:1 depth=0 CN = *.google.com verify return:1 --- Certificate chain 0 s:CN = *.google.com i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3 -----BEGIN CERTIFICATE----- MIIN1jCCDL6gAwIBAgIRAMTFJRnvXaqbEuGoT1RRoNUwDQYJKoZIhvcNAQELBQAw TRIMMED -----END CERTIFICATE----- 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3 i:C = US, O = Google Trust Services LLC, CN = GTS Root R1 -----BEGIN CERTIFICATE----- MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQsw TRIMMED -----END CERTIFICATE----- 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1 i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA -----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBX TRIMMED -----END CERTIFICATE----- --- Server certificate subject=CN = *.google.com issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3 --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: ECDSA Server Temp Key: X25519, 253 bits --- SSL handshake has read 6676 bytes and written 392 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 256 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) ---
Now that the expected certificate chain has been obtained, it can be walked down to check if the certificate is present in the Terraform Enterprise CA certificates. Certificate 1 in Google's chain is their serving certificate for *.google.com`. The i: line details the certificate's issuer, or the CA that signed it. In this case it's Google Trust Services LLC with the GTS CA 1C3 CA. Now, we ensure that the GTS CA 1C3 CA is present in Terraform Enterprise. If it isn't present, add it and try again, otherwise, move to the next certificate in the chain, which is the GTS CA 1C3 CA in this case. It is issued by Google Trust Services LLC with GTS Root R1, which is in turn signed with the GlobalSign nv-sa Root
CA. Each one of these CA certificates in the Terraform Enterprise settings should be in the same order as the chain. Most public CAs' certificates are published publicly, such as Google's and GlobalSign's. For private CAs issued by a business unit in the organization, that team should be able to assist in obtaining the CA certificates.
Additional Information
If the HTTP request takes a long time to reattempt, such as an error during a Terraform apply, the CA certificates from Terraform Enterprise can be exported to a PEM file and used with curl to quickly ensure SSL verification. To do this, simply export the CA certificates from Terraform Enterprise with replicatedctl
app-config export --template '{{.ca_certs.Value}}' > tfe-certs.pem. Then curl can be used to test the HTTP request: curl -v --cacert tfe-certs.pem --capath . https://example.com. This curl command will error with x509: certificate signed by unknown authority errors if the CA certificates aren't sufficient to verify the certificate chain.