Terraform Enterprise - Node drain fails – HashiCorp Help Center

Problem

While running Terraform Enterprise v202311-1 or later in non-consolidated services mode, the following error occurs while attempting to node drain:

root@ip-redacted:/var/snap/amazon-ssm-agent/7666# tfe-admin node-drain
Running node-drain (localhost)
2023-12-11T19:18:12.142Z [INFO] draining node: node=localhost
2023-12-11T19:18:12.143Z [INFO] stopping sidekiq
2023-12-11T19:18:13.044Z [INFO] successfully stopped sidekiq: output="tfe-sidekiq
"
2023-12-11T19:18:13.044Z [INFO] stopping build_manager and build_worker
2023-12-11T19:18:13.084Z [ERROR] error stopping build_manager and build_worker: error="exit status 1"
2023-12-11T19:18:13.084Z [ERROR] Error response from daemon: No such container: tfe-build-manager
Error response from daemon: No such container: tfe-build-worker
: error="exit status 1"
error draining node: error stopping build_manager and build_worker: exit status 1

Cause

The node drain on this version fails due to the removal of the environment variable RUN_PIPELINE which is needed to determine the appropriate method to drain the node.

Overview of possible solution

The solution to this issue is to switch to consolidated services mode and use the new node drain command.

Solution:

Switch to consolidated services mode by running the following command:
replicatedctl app-config set consolidated_services_enabled --value 1
Restart the application
replicatedctl app stop
replicatedctl app start
Perform the node drain with the following command
docker exec terraform-enterprise tfectl node drain

Outcome:

After switching to consolidated services mode and using the new command, the node drain should be able to complete successfully.

Additional Information

If you encounter issues after trying the solution above, please reach out to HashiCorp Support for assistance.

References