Terraform Enterprise displays x509: certificate is not valid for any names or x509: certificate is valid for (Replicated)

Introduction

Workspaces in Terraform Enterprise are configured to execute Terraform runs. During a run, Terraform CLI will communicate with the API of Terraform Enterprise itself as described in the discovery process documentation. Terraform Enterprise executes these Terraform runs in an ephemeral container either using the default worker image or a custom worker image. According to the reference architectures, the fully qualified domain name of Terraform Enterprise is resolved into the load-balancer’s IP address which resulting in the route of the network traffic is from the ephemeral container to the load-balancer for the service discovery process. The hairpin addressing feature has been introduced into Terraform Enterprise in the release v202011-1 to bypass the traffic routing from the ephemeral container to the load-balancer, and instead routing the traffic to the private IP address of the Terraform Enterprise host machine.

Problem

The Terraform Enterprise application serves the secure traffic on port 443 and shares TLS certificates with the Replicated management stack in which operates on port 8800. There are chances that a Terraform run fails with the following error when enable the hairpin addressing feature:

Terraform v0.13.5
Configuring remote state backend...
Initializing Terraform configuration...

Setup failed: Failed terraform init (exit 1): <nil>

Output:

Initializing the backend...

Error: Failed to request discovery document: Get "https://tfe.example.com/.well-known/terraform.json": x509: certificate is not valid for any names, but wanted to match tfe.example.com

  on zzz_backend_override.tf.json line 5, in terraform[0].backend[0].remote:
   5:                 "hostname": "tfe.example.com",

or

Terraform v0.13.5
Configuring remote state backend...
Initializing Terraform configuration...

Setup failed: Failed terraform init (exit 1): <nil>

Output:

Initializing the backend...

Error: Failed to request discovery document: Get "https://tfe.example.com/.well-known/terraform.json": x509: certificate is valid for tfe-test.example.com, not tfe.example.com

  on zzz_backend_override.tf.json line 5, in terraform[0].backend[0].remote:
   5:                 "hostname": "tfe.example.com",

Cause

The Replicated management stack is responsible for generating the self-signed certificate which is used by both the Replicated management console and the Terraform Enterprise application. The fully qualified domain name can be missing or mismatched from the server certificate Subject when it is not specified at the Replicated management layer or specified incorrectly and causing the handshake error to the ephemeral container. The mismatch hostname configuration can occur during the installation time or post-installation time.

Installation

There are two options for the installation mode in Terraform Enterprise, manual installation, and automated installation.

• In the manual installation mode, the hostname can be omitted from the UI input or incorrectly specified.
• In the automated installation mode, the property TlsBootstrapHostname can be omitted or incorrectly specified from the /etc/replicated.conf bootstrapping file.

Post-installation

At the post-installation time, when updating the management hostname on the Replicated layer, omitting or incorrectly specifying the management hostname can also lead to the symptom.

Solution

Installation

During installation, this problem can be solved by the following options in accordance with the installation type:

• Manual/Web browser UI

  The installation Web UI will prompt you to provide the hostname, this is required for the hairpin addressing with the self-signed certificate.

  

• Automated installation

  The bootstrap file /etc/replicated.conf for the automated installation requires to have TlsBootstrapHostname property when the TlsBootstrapType property is set to self-signed with the hairpin addressing enabled.

  {
      "DaemonAuthenticationType": "password",
      "DaemonAuthenticationPassword": "your-password-here",
      "TlsBootstrapType": "self-signed",
      "TlsBootstrapHostname": "server.company.com",
      "BypassPreflightChecks": true,
      "ImportSettingsFrom": "/path/to/settings.json",
      "LicenseFileLocation": "/path/to/license.rli"
  }

Post-installation

Post-installation, this problem can be solved by following the steps below in order to update the hostname:

1. Login to the Replicated console at https://<<TFE FQDN>>:8800.

2. Click on the gear icon on the top right corner.

3. Select Console Settings.

4. Update the Management Console Hostname with the fully qualified name of the Terraform Enterprise application.

   

5. Scroll down to the bottom and click Save.

6. Restart Terraform Enterprise application:

   • Navigating to the Dashboard tab.
   • Click Stop Now and wait until the application fully stopped.
   • Click on Start Now and wait until the application fully started.

Notes

If the Terraform Enterprise is configured with a HTTP proxy, the Proxy Bypass requires to include the fully qualified domain name of the Terraform Enterprise application to route the traffic internally to the private IP address of the Terraform Enterprise host machine without the outgoing traffic to the HTTP proxy when the hairpin addressing is enabled.

Additional information

Please visit the article Hairpin Addressing for how to enable the feature.

If you continue to experience the issues after following the guides, please contact HashiCorp Support to request for further assistance.