Terraform Enterprise: Vault startup failure after upgrading to v202411-1 through v202502-1

Problem

After upgrading Terraform Enterprise, the application fails to start. The /var/log/terraform-enterprise/tfe-vault.log file contains errors indicating a problem with the Vault unseal key.

An example error log is shown below.

{
  "@level": "error",
  "@message": "error returned from run()",
  "@module": "vault-manager",
  "err": "1 error occurred:\n\t* recovered from panic: error unsealing vault: error unsealing vault: Error making API request.\n\nURL: PUT http://127.0.0.1:8200/v1/sys/unseal\nCode: 400. Errors:\n\n* invalid key: key is longer than maximum 33 bytes\n..."
}

You may also see logs indicating the Vault service is shutting down unexpectedly.

==> Vault shutdown triggered
INFO exited: vault (exit status 1; not expected)

Prerequisites

This issue affects users upgrading from a previous version of Terraform Enterprise to one of the following versions:

• v202411-1
• v202412-1
• v202501-1
• v202502-1

Cause

This issue is caused by a defect in Terraform Enterprise versions v202411-1 through v202502-1 that mishandles the Vault unseal key if it is stored in hex format.

Solution

To resolve this issue, you must convert the hex-formatted unseal key to base64 and update it within the Terraform Enterprise container.

1. Access the Terraform Enterprise container by following the Terraform Enterprise CLI access instructions.

2. Once you have a shell inside the container, retrieve the current unseal key. This command sources the necessary environment variables before running the keymgmt tool.

   $ . /run/terraform-enterprise/vault/env; keymgmt get unseal

3. The output will be in either hex or base64 format. Identify the format of your key.
   • Hex format:951dd0d632c0654c1b7e654f81dcd8973dce57b675b0559fef5ba1cc52768db
   • Base64 format:NXX7iBYvMCtEMrmHqXXYU3SDML1AYzP+8rKDaClR6pM=

4. If your key is in hex format, convert it to base64. Replace the example hex value with your own.

   $ echo -n '951dd0d632c0654c1b7e654f81dcd8973dce57b675b0559fef5ba1cc52768db' | xxd -r -p | base64

   The command will output the base64-encoded version of the key. Copy this value.

   NXX7iBYvMCtEMrmHqXXYU3SDML1AYzP+8rKDaClR6pM=

5. Set the unseal key to the new base64 value. This command also requires the root token, which is retrieved dynamically.

   $ keymgmt set <base64_unseal_value> $(keymgmt get root)

6. Verify that the newly encoded value is in place.

   $ keymgmt get unseal

7. Restart all Terraform Enterprise services to apply the change.

   $ supervisorctl restart all

Outcome

After restarting the services, Terraform Enterprise will start up successfully without further Vault errors related to the unseal key.