Introduction
This article provides more detailed instructions on how to update the encryption password for Terraform Enterprise Kubernetes deployments.
Expected Outcome
Once all steps are completed, the application should be up and running with the updated password.
Prerequisites (if applicable)
Procedure
Get the Terraform Enterprise pod name.
kubectl get pods -n terraform-enterprise
NAME READY STATUS RESTARTS AGE
terraform-enterprise-79db4fcdb7-wf448 1/1 Running 0 5dStart a shell in the pod.
kubectl exec -n terraform-enterprise -it terraform-enterprise-79db4fcdb7-wf448 -- bash
Verify the current password using tfectl.
terraform-enterprise@terraform-enterprise-79db4fcdb7-wf448:/$ tfectl app config --unredacted | grep "encryption_password"
"encryption_password": "Password123"
Update the encryption password, making sure to also update the TFE_ENCRYPTION_PASSWORD in the Deployment/ConfigMap, as directed, by creating a new Helm release with the updated setting.
terraform-enterprise@terraform-enterprise-79db4fcdb7-wf448:/$ tfectl app rotate-encryption-password
WARNING: this operation is irreversible, and you will need to restart all of the TFE nodes once this operation is done. Make sure that no one is using TFE at this time.
Do you desire to continue? 'yes' is the only valid option. [yes/no]: yes
Current Encryption Password: ***********
New Encryption Password: **********
Encryption key successfully rotated
Current TFE_ENCRYPTION_PASSWORD value in the Kubernetes deployment/config map is not up to date.
Update it with the new encryption password and rotate the pods.Drain the node(s) to stop the them from picking up any new tasks. The example given is a single node instance. If you have multiple, use the --all flag.
terraform-enterprise@terraform-enterprise-79db4fcdb7-wf448:/$ tfectl node drain
stopping service: service=sidekiq
successfully stopped service: service=sidekiq
stopping service: service=task-worker
successfully stopped service: service=task-worker
node successfully drained: node=terraform-enterprise-79db4fcdb7-wf448
Exit the Terraform Enterprise Pod
terraform-enterprise@terraform-enterprise-79db4fcdb7-wf448:/$ exit
exit
command terminated with exit code 1Restart the Terraform Enterprise pod. If all steps were followed, a new Terraform Enterprise pod should come up.
kubectl get pods -n terraform-enterprise Troubleshooting
If the new pod fails to enter a healthy state, use the following commands to review the pod logs for more information.
Get the pod name.
kubectl get pods -n terraform-enterprise
NAME READY STATUS RESTARTS AGE
terraform-enterprise-79db4fcdb7-z87t8 0/1 Running 0 131m
Check logs for error messages. In this example, there is an error message indicating the Vault unseal key could not be decrypted using the current encryption password in the application configuration. The error is caused by failing to update TFE_ENCRYPTION_PASSWORD in the Kubernetes Deployment/ConfigMap with the new value.
kubectl logs terraform-enterprise-79db4fcdb7-z87t8 -n terraform-enterprise | tail
{"log":"10.0.172.152 - - [06/Mar/2024:20:36:16 +0000] \"GET /_health_check HTTP/1.1\" 502 150 \"-\" \"kube-probe/1.28+\"","component":"nginx"}
{"log":"Waiting for Atlas to become active.","component":"task-worker"}
{"log":"Error reading Vault configuration: failed decrypting unseal key: could not decrypt ciphertext: chacha20poly1305: message authentication failed","component":"vault"}
{"log":"Waiting to retrieve Vault unseal key.","component":"vault"}
{"log":"2024/03/06 20:36:17 [error] 496#496: *1767 connect() failed (111: Unknown error) while connecting to upstream, client: 10.0.172.152, server: , request: \"GET /_health_check HTTP/1.1\", upstream: \"http://127.0.0.1:9292/_health_check\", host: \"10.0.160.173:8080\"","component":"nginx"}
{"log":"10.0.172.152 - - [06/Mar/2024:20:36:17 +0000] \"GET /_health_check HTTP/1.1\" 502 150 \"-\" \"kube-probe/1.28+\"","component":"nginx"}
{"log":"Error checking seal status: Get \"http://127.0.0.1:8200/v1/sys/seal-status\": dial tcp 127.0.0.1:8200: connect: connection refused","component":"archivist"}
{"log":"Waiting for Vault to unseal.","component":"archivist"}
{"log":"Error checking seal status: Get \"http://127.0.0.1:8200/v1/sys/seal-status\": dial tcp 127.0.0.1:8200: connect: connection refused","component":"backup-restore"}
{"log":"Waiting for Vault to unseal.","component":"backup-restore"}