Introduction
When attempting to start the Terraform Enterprise application, it may fail to start and display an error in the installer dashboard at https://$TFE_HOSTNAME:8800 that reads: “Some containers have stopped unexpectedly.”
Problem
One possible cause of this is an issue with the RabbitMQ container. If this is the case, the logs for the Docker container will show the following error.
Error: amqp://rabbitmq:5672: not ready yet: Exception (403) Reason: "username or password not allowed"
PLAIN login refused: user 'hashicorp' - invalid credentials
To review the RabbitMQ logs, connect to the instance using SSH and run the following command.
$ docker logs rabbitmq
Cause
This issue will occur when Replicated is reinstalled over an existing Terraform Enterprise installation and the existing rabbitmq docker volume was not deleted prior to the reinstall. The volume deletion can be done either manually or with a docker restart while the TFE application is down.
A new random rabbitmq password will be generated during each Replicated installation yet the oldrabbitmqcredentials are cached on the rabbitmq docker volume so login to rabbitmq will fail during TFE startup. This will be the case even if the rabbitmq container itself is reporting the updated rabbitmq password.
Solution
In order to resolve this error the rabbitmq volume will need to be removed using the following command:
$ replicatedctl app stop
$ docker volume rm rabbitmq
Once the volume has been removed, attempt to start Terraform Enterprise by using the following command:
$ replicatedctl app start
To verify the application state run replicatedctl app status to monitor the state. Terraform Enterprise should be accessible once the state has transitioned to started.
If you continue to experience issues after removing the rabbitmq volume, please contact HashiCorp support.