Problem
In Terraform Enterprise (TFE) Flexible Deployment Options (FDO) v1.1.0, Terraform runs that use remote agents may remain in a pending state and fail with x509: certificate signed by unknown authority errors. This issue prevents agents from registering with Terraform Enterprise and blocks all runs that rely on remote agents.
The agent logs may show an error similar to the following.
{
"@level": "info",
"@message": "[ERROR] agent: Failed starting core plugin: error=\"failed configuring core: agent registration failed: POST https://ccoe-tf-ent.hii-tsd.com/api/agent/register giving up after 1 attempt(s): Post \"https://ccoe-tf-ent.hii-tsd.com/api/agent/register\": tls: failed to verify certificate: x509: certificate signed by unknown authority\"",
"@module": "task-worker.executor.task-output",
"id": "b5cc9a20-3c2a-49f5-8402-fceb7c2fc658",
"name": "agent-run",
"organization_name": "CCOE",
"run_id": "run-EGoATLGF5wvomhEp",
"stream": "stdout",
"workload_id": "plan-iSybzafGctwvx2ju",
"workload_type": "Plan",
"workspace_name": "aws-security-comm"
}Prerequisites
This issue affects environments with the following characteristics:
- Terraform Enterprise FDO v1.1.0 or v1.1.1.
- A custom CA Bundle is in use.
- Workflows depend on remote agents.
- Environments using self-signed or custom internal CAs are most impacted.
Cause
A bug in the agent image build process in TFE v1.1.0 causes incorrect file permissions on the CA certificate bundle inside the agent container. This prevents the agent from correctly reading the CA bundle, leading to TLS certificate verification failures during registration.
Solutions
The permanent solution is to upgrade to a patched version of Terraform Enterprise where this issue is resolved. The following workaround can be used to temporarily restore functionality.
Workaround: Manually Rebuild the Agent Image
You can manually rebuild the agent image using the Docker CLI to correct the file permissions on the CA certificate bundle. This action is performed from within the Terraform Enterprise container and rebuilds the hashicorp/tfe-agent:now image using the existing task-worker context.
Important Considerations:
- This is a temporary workaround and not a permanent solution.
- You must repeat this process any time the agent image is regenerated, such as during an application restart.
Execute the following command to rebuild the container.
$ docker exec -u0 -ti "terraform-enterprise-tfe-1" sh -c 'tar -C /run/terraform-enterprise/task-worker/docker -cf - . | curl \ --unix-socket /var/run/docker.sock \ -H "Content-Type: application/x-tar" \ --data-binary @- \ "http:/v1.43/build?t=hashicorp/tfe-agent:now"'
Outcome
After successfully rebuilding the agent image, the CA bundle will have the correct permissions inside the container. Remote agents will be able to register with Terraform Enterprise, and remote-agent–based runs will proceed as expected.
Additional Information
- For details on the official patch, refer to the Terraform Enterprise v1.1.x release notes.