Problem
Terraform Enterprise fails to start with the following log output from the vault-manager service.
{"@level":"info","@message":"starting vault-manager","@module":"vault-manager","@timestamp":"2024-12-20T23:43:46.700217Z","refresh only":false}
{"@level":"info","@message":"starting vault","@module":"vault-manager","@timestamp":"2024-12-20T23:43:46.700307Z"}
{"@level":"info","@message":"vault","@module":"vault-manager","@timestamp":"2024-12-20T23:43:46.700811Z","pid":397}
panic: error running keymgmt get unseal: exit status 1
goroutine 1 [running]:
main.main()
/home/runner/work/terraform-enterprise/terraform-enterprise/tools/vault-manager/main.go:104 +0x5e8
Prerequisites
- Terraform Enterprise >= v202411-1
- Terraform Enterprise is configured to use internal Vault
Cause
This startup error is typically caused by an incorrect encryption password in the Terraform Enterprise configuration. Terraform Enterprise requires an encryption password which is used to encrypt and decrypt the root token and unseal key of the internally managed Vault. If the value is changed in the configuration without following the proper rotation procedure, Terraform Enterprise will be unable to decrypt these artifacts and set up Vault.
To confirm that this is the cause, run the following command to manually execute the internal keymgmt utility from a shell in the Terraform Enterprise container.
- Docker
docker exec -ti <TFE_CONTAINER> bash -c '. /run/terraform-enterprise/vault/env; keymgmt get unseal'
- Podman
podman exec -ti <TFE_CONTAINER> bash -c '. /run/terraform-enterprise/vault/env; keymgmt get unseal'
- Kubernetes/OpenShift
kubectl exec -ti <TFE_POD> -- bash -c '. /run/terraform-enterprise/vault/env; keymgmt get unseal'
If the configured encryption password does not match that which was used to encrypt the Vault artifacts, the output of the keymgmt command will be the following.
Error reading Vault configuration: failed decrypting unseal key: could not decrypt ciphertext: chacha20poly1305: message authentication failed
Solution
Replace the encryption password in the Terraform Enterprise configuration with the previously set value and redeploy the application.
For a Replicated deployment, run the following command to change the setting, or modify the configuration in the automated installation script and repave the nodes.
replicatedctl app-config set enc_password --value '<PASSWORD>'
replicatedctl app apply-config
For a Flexible Deployment Options deployment, modify the TFE_ENCRYPTION_PASSWORD setting. Consult the Terraform Enterprise Flexible Deployment Options documentation for the respective deployment option for details on where to change the setting.
If the encryption password appears to be unchanged and matches the previous password, consult the documentation linked below to troubleshoot whether or not the value is being read in correctly by the underlying container platform.