Tracing SSL certificate chain issues in Terraform Enterprise

Problem

When Terraform Enterprise attempts to make an outbound HTTPS request during installation, within container logs, or during a Terraform plan or apply, the operation fails with an error similar to the following.

x509: certificate signed by unknown authority

The error is typically prefixed by the URL of the failed request. This indicates that Terraform Enterprise was unable to verify one or more of the Certificate Authorities (CAs) that signed the SSL certificates served by that URL.

Cause

This error occurs because the list of PEM-formatted CA certificates configured in Terraform Enterprise is either missing a required intermediate or root certificate, or the certificates are not in the correct hierarchical order. Terraform Enterprise containers use this list to verify SSL certificates from private or corporate Certificate Authorities.

Solutions

Follow these solutions to diagnose and resolve the certificate chain issue.

Solution 1: Verify Certificate Order and Format

Before proceeding with manual tracing, ensure that the CA certificates in your Terraform Enterprise settings are correctly formatted and ordered. The certificate for the signing authority should appear after the certificate it signed.

For detailed guidance on the required format and order, refer to the Terraform Enterprise: Order of certificates in SSL cert files article.

Solution 2: Manually Trace and Verify the Certificate Chain

If the order and format are correct, you can use openssl to inspect the certificate chain served by the remote endpoint and verify that each certificate in the chain is present in your Terraform Enterprise configuration.

This example uses google.com for demonstration purposes.

1. Retrieve the certificate chain from the target endpoint. The output shows the chain of certificates, from the server certificate (depth 0) to the root.

   $ openssl s_client -showcerts -connect google.com:443 < /dev/null

   A trimmed version of the output appears below.

     CONNECTED(00000198)
     depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
     verify return:1
     depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
     verify return:1
     depth=0 CN = *.google.com
     verify return:1
     ---
     Certificate chain
     0 s:CN = *.google.com
     i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
     -----BEGIN CERTIFICATE-----
     MIIN1jCCDL6gAwIBAgIRAMTFJRnvXaqbEuGoT1RRoNUwDQYJKoZIhvcNAQELBQAw
     TRIMMED
     -----END CERTIFICATE-----
     1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
     i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
     -----BEGIN CERTIFICATE-----
     MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQsw
     TRIMMED
     -----END CERTIFICATE-----
     2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
     i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
     -----BEGIN CERTIFICATE-----
     MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBX
     TRIMMED
     -----END CERTIFICATE-----
     ---
     Server certificate
     subject=CN = *.google.com
     issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
     ---
     No client certificate CA names sent
     Peer signing digest: SHA256
     Peer signature type: ECDSA
     Server Temp Key: X25519, 253 bits
     ---
     SSL handshake has read 6676 bytes and written 392 bytes
     Verification: OK
     ---
     New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
     Server public key is 256 bit
     Secure Renegotiation IS NOT supported
     Compression: NONE
     Expansion: NONE
     No ALPN negotiated
     Early data was not sent
     Verify return code: 0 (ok)
     ---

2. Analyze the certificate chain. The s: line indicates the certificate's subject, and the i: line indicates its issuer (the CA that signed it).
   • Certificate 0 (*.google.com) is issued by GTS CA 1C3.
   • Certificate 1 (GTS CA 1C3) is issued by GTS Root R1.
   • Certificate 2 (GTS Root R1) is issued by GlobalSign Root CA.
3. Verify that each issuer certificate exists in your Terraform Enterprise CA bundle. Walk down the chain and confirm that the certificate for GTS CA 1C3, GTS Root R1, and GlobalSign Root CA are all present in your TFE settings and in the correct order.
4. Add any missing CA certificates to your Terraform Enterprise configuration. For public CAs, you can typically download the certificates from the authority's website. For private CAs, contact the internal team that manages them to obtain the correct certificates.

Additional Information

Quick Verification with cURL

If troubleshooting a long-running process like a Terraform apply, you can test your CA bundle more quickly using curl.

1. Export the currently configured CA certificates from Terraform Enterprise to a local file named tfe-certs.pem.

   $ replicatedctl app-config export --template '{{.ca_certs.Value}}' > tfe-certs.pem

2. Use curl to test the HTTPS request, pointing to your exported CA bundle. If the certificates are insufficient to verify the chain, this command will fail with the same x509 error.

   $ curl -v --cacert tfe-certs.pem https://example.com