Problem
When using Terraform Enterprise in consolidated services mode, you may encounter a critical issue where the Atlas service fails to start. This problem occurs because the AMPQ_URL environment variable is not set in the terraform-enterprise container, which can cause connection errors in specific network configurations.
Prerequisites
- Terraform Enterprise v202309-1 or later in a Replicated deployment using consolidated services mode.
- This issue may also affect Docker Flexible Deployment Option installations.
Cause
In Terraform Enterprise's consolidated services mode, the AMPQ_URL environment variable is not set in the terraform-enterprise container because a RabbitMQ service is no longer required. However, the Atlas component attempts to connect to rabbitmq:5672 by default during startup.
In most installations, the absence of the rabbitmq hostname results in a harmless DNS resolution warning, and Atlas starts successfully. However, if your corporate network runs a private RabbitMQ service and your DNS is configured to search a local domain (e.g., mycompany.com), a hostname of rabbitmq may resolve to an internal service like rabbitmq.mycompany.com. This can cause Atlas to attempt an unauthorized connection, resulting in a non-recoverable authentication error.
The following error appears in the logs.
ERROR -- #Bunny::Session:0x1f5e0 hashicorp@rabbitmq:5672, vhost=hashicorp, addresses=[rabbitmq:5672]>: Authentication with RabbitMQ failed: 403 ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.","component":"atlas"}
Solutions
Solution 1: Remove the Search Entry in /etc/resolv.conf
A viable workaround is to modify the /etc/resolv.conf file on your Terraform Enterprise host to eliminate the search entry that resolves to the problematic hostname.
This solution is suitable if no other services on the host rely on that specific DNS search configuration. Note that this change might not be effective for all scenarios.
Solution 2: Upgrade to Terraform Enterprise v202312-1
This has been resolved in v202312-1.