Terraform Enterprise Fails to Start With "panic: error running keymgmt get unseal: exit status 1" – HashiCorp Help Center

Problem

Terraform Enterprise fails to start, and the vault-manager service logs show a panic error.

{
  "@level": "info",
  "@message": "starting vault-manager",
  "@module": "vault-manager",
  "@timestamp": "2024-12-20T23:43:46.700217Z",
  "refresh only": false
}
{
  "@level": "info",
  "@message": "starting vault",
  "@module": "vault-manager",
  "@timestamp": "2024-12-20T23:43:46.700307Z"
}
{
  "@level": "info",
  "@message": "vault",
  "@module": "vault-manager",
  "@timestamp": "2024-12-20T23:43:46.700811Z",
  "pid": 397
}
panic: error running keymgmt get unseal: exit status 1

goroutine 1 [running]:
main.main()/home/runner/work/terraform-enterprise/terraform-enterprise/tools/vault-manager/main.go:104 +0x5e8

Prerequisites

Terraform Enterprise version v202411-1 or newer.
Terraform Enterprise is configured to use the internal Vault service.

Cause

This startup error typically occurs when an incorrect encryption password is present in the Terraform Enterprise configuration. Terraform Enterprise uses this password to encrypt and decrypt the root token and unseal key for its internally managed Vault instance. If this password is changed in the configuration without following the proper rotation procedure, Terraform Enterprise cannot decrypt these critical artifacts and fails to start Vault.

Verifying the Cause

To confirm that an incorrect password is the cause, you can manually execute the internal keymgmt utility from a shell inside the Terraform Enterprise container.

For Docker deployments:

$ docker exec -ti <TFE_CONTAINER> bash -c '. /run/terraform-enterprise/vault/env; keymgmt get unseal'

For Podman deployments:

$ podman exec -ti <TFE_CONTAINER> bash -c '. /run/terraform-enterprise/vault/env; keymgmt get unseal'

For Kubernetes or OpenShift deployments:

$ kubectl exec -ti <TFE_POD> -- bash -c '. /run/terraform-enterprise/vault/env; keymgmt get unseal'

If the configured encryption password does not match the one used to encrypt the Vault artifacts, the command will fail with the following output.

Error reading Vault configuration: failed decrypting unseal key: could not decrypt ciphertext: chacha20poly1305: message authentication failed

Solutions

Solution 1: Restore the Previous Encryption Password

Replace the incorrect encryption password in your Terraform Enterprise configuration with the previously correct value and redeploy the application.

For Replicated deployments:

Update the setting using replicatedctl commands. Alternatively, modify the configuration in your automated installation script and re-pave the nodes.
```
$ replicatedctl app-config set enc_password --value '<PASSWORD>'
$ replicatedctl app apply-config
```
For Flexible Deployment Options (FDO) deployments:

Modify the TFE_ENCRYPTION_PASSWORD setting in your environment configuration. Consult the Terraform Enterprise Flexible Deployment Options documentation for your specific deployment method for details on where to change this setting.

Solution 2: Troubleshoot Password Sourcing

If you have confirmed the encryption password is correct in your configuration files but the error persists, the value may not be correctly read by the underlying container platform. Refer to the linked documentation to troubleshoot this scenario further.

Additional Information