Terraform Runs are Getting Stuck on Cost Estimation/Policy checks

Avatar
Darrell F. Williams
  • Updated

Problem

After updating the SSL certificate for Terraform Enterprise (TFE) and correctly applying the new certificate to all external Terraform agents, workspace runs began to fail during the Cost Estimation phase. Notably, the Plan phase continues to complete successfully.

The affected workspaces are configured to use external agents for execution.

Upon reviewing the atlas logs on the TFE instance, the following SSL-related error was observed:

{:error=>"RestClient::SSLCertificateNotVerified", :id=>12345, :message=>"SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)"}

This indicates that the Cost Estimation service attempted to establish a secure connection but failed due to an expired or untrusted certificate, despite the new certificate being in place on the TFE agents.

Prerequisites

  • Workspace runs execution mode is set to Agents
  • Cost Estimation or Sentinel is enabled on the workspace

Cause

  • Despite the workspace being set to use external agents, the Cost Estimation step is handled by Terraform Enterprise’s internal agents. These internal agents rely on the bundle.pem file on the TFE server to validate SSL connections.

  • While the new SSL certificate was correctly applied to the external agents, it was not updated in the bundle.pem on the TFE host. As a result, the internal agents failed SSL verification during Cost Estimation due to referencing an expired certificate.

Solutions:

To resolve the SSL verification issue during the Cost Estimation phase, the updated certificate must be added to the bundle.pem file on the TFE server. This ensures that TFE’s internal agents can validate SSL connections correctly.

Follow the appropriate steps below based on your TFE installation method to update the bundle.pem file with the new cert.

  1. For updating the bundle.pem on the TFE Flexible Deployment Option (FDO) such as the Docker deployment in accordance to these cert requirements please follow Updating Certificates in Terraform Enterprise (TFE) Docker Based Installation.
  2.  To Update the bundle.pem file on the Replicated release, please assess this nifty article, How to setup Terraform Enterprise with a Certificate Authority (CA) Bundle.

Outcome

After the bundle.pemhas been updated, the Cost Estimation or Sentinel policy checks phase in execution should process successfully 

Additional Information

Articles in this section