Terraform Enterprise Restore Fails With Encryption Key Not Found – HashiCorp Help Center

Problem

When migrating from an externally managed Vault service to the internal Vault service provided with Terraform Enterprise, the backup process may not capture all required keys. This causes the subsequent restore process to fail with an encryption key not found error.

Prerequisites

Terraform Enterprise configured with an external HashiCorp Vault instance.

Cause

The default Terraform Enterprise Vault policy restricts access to certain data paths required to export the Vault transit keys. During a backup, this permission denial prevents the archivist and atlas keys from being included in the snapshot, leading to a restore failure.

The following error may appear in the backup container logs, indicating a permission issue:

transit/handler: error transmitting backup: error="error populating atlas data in snapshot: Error making API request.URL: GET https://$vaultinstance:8200/v1/transit/keys/archivist_kdfCode: 403. Errors:* 1 error occurred:* permission deniedtransit/handler: finished sending snapshot to HTTP response

Note: For Terraform Enterprise v202205-1 or later, container names have changed. The p prefix has been removed (e.g., ptfe-backup-restore is now tfe-backup-restore).

Solution

To resolve this issue, you must update the Vault policy for Terraform Enterprise to grant read access to the necessary transit key paths.

Check the Docker logs on the original instance for errors related to the archivist or atlas Vault keys.

## For versions before v202205-1
$ sudo docker logs ptfe-backup-restore

## For versions v202205-1 and later
$ sudo docker logs tfe-backup-restore

Create a backup of your tfe.hcl policy file before modifying it.
```
$ cp ./tfe.hcl ./tfe.bkup
```
Connect to the external Vault instance via SSH and log in to Vault with appropriate credentials.
```
$ vault login
```

Modify the tfe.hcl policy file to add read capabilities to the archivist and atlas transit paths.

# Add "read" to this rule
path "transit/keys/archivist_*" {
  capabilities = ["read", "update", "create"]
}

# Add "read" to this rule
path "transit/backup/archivist_*" {
  capabilities = ["read", "create", "update"]
}

# Add "read" to this rule
path "transit/keys/atlas_*" {
  capabilities = ["read", "update", "create"]
}

# Add this policy rule
path "transit/backup/atlas_*" {
  capabilities = ["read"]
}

After saving the file, write the updated policy to Vault.
```
$ vault policy write tfe tfe.hcl
```
Verify that the tfe policy contains the modified permissions.
```
$ vault policy read tfe
```

Run the backup API request again.

$ curl \
  --header "Authorization: Bearer $TOKEN" \
  --request POST \
  --data @payload.json \
  --output backup.blob \
  https://<TFE_HOSTNAME>/_backup/api/v1/backup

Outcome

After applying the updated Vault policy, the backup process will complete successfully, and the resulting snapshot will contain all the necessary keys for a successful restore.

Additional Information