Problem
While running Terraform Enterprise v202311-1
or later in non-consolidated services
mode, the following error occurs while attempting to node drain:
root@ip-redacted:/var/snap/amazon-ssm-agent/7666# tfe-admin node-drain
Running node-drain (localhost)
2023-12-11T19:18:12.142Z [INFO] draining node: node=localhost
2023-12-11T19:18:12.143Z [INFO] stopping sidekiq
2023-12-11T19:18:13.044Z [INFO] successfully stopped sidekiq: output="tfe-sidekiq
"
2023-12-11T19:18:13.044Z [INFO] stopping build_manager and build_worker
2023-12-11T19:18:13.084Z [ERROR] error stopping build_manager and build_worker: error="exit status 1"
2023-12-11T19:18:13.084Z [ERROR] Error response from daemon: No such container: tfe-build-manager
Error response from daemon: No such container: tfe-build-worker
: error="exit status 1"
error draining node: error stopping build_manager and build_worker: exit status 1
Cause
The node drain on this version fails due to the removal of the environment variable RUN_PIPELINE
which is needed to determine the appropriate method to drain the node.
Overview of possible solution
The solution to this issue is to switch to consolidated services
mode and use the new node drain command.
Solution:
-
Switch to consolidated services mode by running the following command:
replicatedctl app-config set consolidated_services_enabled --value 1
- Restart the application
replicatedctl app stop
replicatedctl app start
- Perform the node drain with the following command
docker exec terraform-enterprise tfectl node drain
Outcome:
After switching to consolidated services mode and using the new command, the node drain should be able to complete successfully.
Additional Information
If you encounter issues after trying the solution above, please reach out to HashiCorp Support for assistance.
References
- https://developer.hashicorp.com/terraform/enterprise/replicated/administration/infrastructure/consolidated-services
- https://developer.hashicorp.com/terraform/enterprise/flexible-deployments/admin/admin-cli/admin-cli#gracefully-stop-work-on-a-node