Terraform Enterprise: Vault startup failure after upgrading to v202411-1 through v202502-1 – HashiCorp Help Center

Problem

After upgrading Terraform Enterprise, the application fails to start with errors in /var/log/terraform-enterprise/tfe-vault.log

{"@level":"error","@message":"error returned from run()","@module":"vault-manager","@timestamp":"2025-03-04T05:43:36.980420Z","err":"1 error occurred:\n\t* recovered from panic: error unsealing vault: error unsealing vault: Error making API request.\n\nURL: PUT http://127.0.0.1:8200/v1/sys/unseal\nCode: 400. Errors:\n\n* invalid key: key is longer than maximum 33 bytes\ngoroutine 7 [running]:\nruntime/debug.Stack()\n\t/opt/hostedtoolcache/go/1.23.6/x64/src/runtime/debug/stack.go:26 +0x5e\nmain.(*manager).run.func1()\n\t/home/runner/work/terraform-enterprise/terraform-enterprise/tools/vault-manager/main.go:177 +0x7d\npanic({0x94eb00?, 0xc00052a0f0?})\n\t/opt/hostedtoolcache/go/1.23.6/x64/src/runtime/panic.go:785 +0x132\nmain.(*manager).run(0xc00021a9a0, 0xc00004f5e0)\n\t/home/runner/work/terraform-enterprise/terraform-enterprise/tools/vault-manager/main.go:229 +0x4ee\nmain.main.func1()\n\t/home/runner/work/terraform-enterprise/terraform-enterprise/tools/vault-manager/main.go:139 +0x25\ncreated by main.main in goroutine 1\n\t/home/runner/work/terraform-enterprise/terraform-enterprise/tools/vault-manager/main.go:138 +0x599\n\n\n"}

{"component":"vault","log":"{\"@level\":\"debug\",\"@message\":\"sending SIGINT to vault\",\"@module\":\"vault-manager\",\"@timestamp\":\"2025-02-21T16:40:54.235969Z\",\"pid\":847}"}
{"component":"vault","log":"==\u003e Vault shutdown triggered"}
{"component":"vault","log":"{\"@level\":\"debug\",\"@message\":\"clearing encrypted token from redis\",\"@module\":\"vault-manager\",\"@timestamp\":\"2025-02-21T16:40:54.269350Z\",\"key\":\"vault:token:2c696740e6348d2062d025970d9c293d5b8108e258b4264188a00720b8a00998\"}"}
{"component":"vault","log":"{\"@level\":\"info\",\"@message\":\"goodbye!\",\"@module\":\"vault-manager\",\"@timestamp\":\"2025-02-21T16:40:54.270517Z\",\"errored\":true}"}
2025-02-21 16:40:54,272 INFO exited: vault (exit status 1; not expected)
{"component":"supervisord","log":"2025-02-21 16:40:54,272 INFO exited: vault (exit status 1; not expected)"}

Prerequisites

Upgrading from a previous version of Terraform Enterprise to v202411-1 through v202502-1

Cause

Issue is caused by a defect in v202401-1 through v202502-1 that mishandles the Vault unseal key if stored in hex format.

Solutions:
Follow instructions here to get access the Terraform Enterprise CLI.

Once inside the container, execute the following:

$ . /run/terraform-enterprise/vault/env; keymgmt get unseal

The output will be presented in one of two formats: hex, or base64:

Hex format: 951dd0d632c0654c1b7e654f81dcd8973dce57b675b0559fef5ba1cc52768db
Base64 format: NXX7iBYvMCtEMrmHqXXYU3SDML1AYzP+8rKDaClR6pM=

If the value is hex formatted, convert it into base64 and save it:

$ echo -n '951dd0d632c0654c1b7e654f81dcd8973dce57b675b0559fef5ba1cc52768db'| xxd -r -p | base64

NXX7iBYvMCtEMrmHqXXYU3SDML1AYzP+8rKDaClR6pM=

Copy the base64 encoded value of the unseal key.

Set the key:

$ keymgmt set <base64_unseal_value> $(keymgmt get root)

Now, check that the newly encoded value is in place:

$ keymgmt get unseal

Restart Terraform Enterprise services:

$ supervisorctl restart all

Outcome

Terraform Enterprise startup will succeed without further Vault errors.