Terraform Enterprise startup error - exit status 7

Problem

When starting Terraform Enterprise, the main container tfe-1 repeatedly crashes and fails to initialize. Reviewing the container logs reveals a generic startup error with exit status 7 and indicates the nginx process has exited unexpectedly.

tfe-1 | {"component":"supervisord","log":"2024-10-11 08:10:43,834 INFO exited: nginx (exit status 1; not expected)"}
tfe-1 | {"component":"terraform-enterprise","log":"2024-10-11T08:10:43.824Z [ERROR] terraform-enterprise: startup: error=\"exit status 7\""}

Prerequisites

• Terraform Enterprise

Cause

The Terraform Enterprise container runs multiple supervised processes. The generic exit status 7 error indicates that a critical subprocess has failed. In this case, the logs show that the nginx web server, which serves the application UI and API, is crashing on startup.

The default container logs do not capture the specific reason for the Nginx failure. This issue is commonly caused by a misconfiguration of the provided TLS certificate, such as a mismatch between the private key and the public certificate.

Solution

To resolve this issue, you must retrieve the detailed Nginx error logs from within the container to identify the root cause and then verify your TLS certificate and key.

1. Retrieve Detailed Nginx Logs

The specific Nginx error is available in /var/log/terraform-enterprise/nginx.log inside the container. Because the container is crashing, you must execute the command quickly during the brief window when the container is starting.

For Docker-based deployments, run the following command:

$ docker exec -it terraform-enterprise cat /var/log/terraform-enterprise/nginx.log

For Kubernetes-based deployments, run the following command, replacing NAMESPACE with the appropriate namespace:

$ kubectl -n NAMESPACE exec -it terraform-enterprise -- cat /var/log/terraform-enterprise/nginx.log

2. Identify the Certificate Error

A common error found in the nginx.log is a key value mismatch, which will appear similar to the following output.

nginx: [emerg] SSL_CTX_use_PrivateKey("/etc/ssl/private/terraform-enterprise/key.pem") failed (SSL: error:05800074:x509 certificate routines::key values mismatch)

This error confirms that the provided private key does not correspond to the public certificate.

3. Verify Certificate and Key Match

You can verify if your certificate and private key match by comparing the MD5 hash of their modulus. If the hashes do not match, you must obtain a correct key pair from the team that issued the TLS certificate.

• Example of a matching certificate and key:

  The MD5 hashes in the output are identical.

  $ openssl x509 -noout -modulus -in cert.pem | openssl md5
  ## (stdin)= b3b8253a1e9b8e797f550f53780a8927
  
  $ openssl rsa -noout -modulus -in key.pem | openssl md5
  ## (stdin)= b3b8253a1e9b8e797f550f53780a8927

• Example of a mismatched certificate and key:

  The MD5 hashes in the output are different.

  $ openssl x509 -noout -modulus -in cert.pem | openssl md5
  ## (stdin)= c67e374fcd59f60ab35b75f5cbba1944
  
  $ openssl rsa -noout -modulus -in key.pem | openssl md5
  ## (stdin)= 53489c2da829370f21b77f6b754afea7

Outcome

After replacing the misconfigured TLS certificate and private key with a valid, matching pair, the Nginx process will start successfully. This allows the Terraform Enterprise application to initialize and become available.

Additional Information

• For more details on TLS certificate requirements, refer to the Terraform Enterprise TLS Certificates documentation.