Problem
Users of Terraform Enterprise in consolidated services mode may encounter a critical issue during Atlas startup. This problem relates to the absence of an essential environment variable, AMPQ_URL, in the terraform-enterprise container. While this might not be an issue for most users, it can lead to complications for individuals with specific network configurations.
Prerequisites
- Replicated Deployment of Terraform Enterprise v202309-1 in consolidated services mode
- May also affect Docker Flexible Deployment Option installations of Terraform Enterprise
Cause
In Terraform Enterprise's consolidated services mode, the AMPQ_URL environment variable is not set in the terraform-enterprise container. This omission is typically inconsequential because there is no longer a RabbitMQ service in TFE. However, Atlas, the component responsible for certain operations, attempts to connect to rabbitmq:5672 by default during startup. This behavior differs from previous versions, where it attempted to connect to tfe-rabbitmq:5672.
For standard TFE installations, the absence of either hostname (rabbitmq or tfe-rabbitmq) is generally harmless, as a DNS resolution error only emits a warning and Atlas proceeds to start up. However, for users who run a private RabbitMQ service on their corporate network, which often uses a specific naming convention (e.g., rabbitmq.mycompany.com) with DNS configuration set to search in mycompany.com, this can result in a non-recoverable authentication error.
ERROR -- #Bunny::Session:0x1f5e0 hashicorp@rabbitmq:5672, vhost=hashicorp,
addresses=[rabbitmq:5672]>: Authentication with RabbitMQ failed: 403
ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN.
For details see the broker logfile.","component":"atlas"}
Solution
Users facing the issue of Atlas failing to start in Terraform Enterprise consolidated services mode due to the absence of the AMPQ_URL environment variable can consider the following solution:
-
Remove the Search Entry in
/etc/resolv.conf: A viable workaround is to modify the/etc/resolv.conffile on your TFE Host, to eliminate the search entry that resolves to the problematic hostname. This solution may be suitable for users, especially if there are no additional services on the host relying on that specific DNS configuration. However, please note that it might not be effective for all scenarios.
Outcome
Terraform Enterprise Engineering has been alerted to this issue and is currently in the process of investigating a resolution that will be included in a later release of Terraform Enterprise.
Additional Information
-
If you have additional questions or concerns, please contact HashiCorp Support.