Problem
When migrating from an externally managed Vault service to the provided Terraform Enterprise internal Vault service, the backup process does not capture all the required keys which causes the restore process to fail with an encryption key not found.
Prerequisites
- Hashicorp externally managed Vault
- Hashicorp Terraform Enterprise
Cause
- This is due to the Terraform Enterprise Vault policy restricting access to portions of data needed to export the Vault transit keys.
NOTE: For Terraform Enterprise v202205-1 or later the container names have changed as the "p" has been dropped, such as for example in comparison pre v202205-1 container names would suggest ptfe-backup-restore where v202205-1 and above would reflect tfe-backup-restore.
Solution
1. Check the Docker log on the original instance where the external Vault is located for errors associated with the archivist or atlas Vault keys.
$ sudo docker logs ptfe-backup-restore
$ sudo docker logs tfe-backup-restore (TFE v202205-1 or above)
transit/handler: error transmitting backup: error="error populating atlas data in snapshot:
Error making API request.
URL: GET https://$vaultinstance:8200/v1/transit/keys/archivist_kdf
Code: 403. Errors:
* 1 error occurred:
* permission denied
transit/handler: finished sending snapshot to HTTP response
2. Make a backup of the tfe.hcl file prior to modifying it.
$ cp ./tfe.hcl ./tfe.bkup
3. Connect to the external Vault instance via SSH and login to Vault with the root key.
$ vault login
4. Once authenticated modify the tfe policy file to reflect the below-listed rights so that the archivist and atlas transit paths have read access and save the file.
$ vi tfe.hcl
//add read to this rule
path "transit/keys/archivist_*" {
capabilities = ["read","update","create"]
}
//Add read to this rule
path "transit/backup/archivist_*" {
capabilities = ["read", "create", "update"]
}
//Add read to this rule
path "transit/keys/atlas_*" {
capabilities = ["read","update","create"]
}
//add this policy rule
path "transit/backup/atlas_*" {
capabilities = ["read"]
}5. Once the file is saved rewrite the policy to Vault using the command below.
$ vault policy write tfe tfe.hcl
6. Verify that the tfe policy contains the modified rights.
$ vault policy read tfe.hcl
7. Run the backup restore API request again.
$ curl \ --header "Authorization: Bearer $TOKEN" \ --request POST \ --data @payload.json \ --output backup.blob \ https://<TFE HOSTNAME>/_backup/api/v1/backup
You should now have a successful backup from the original external Vault instance after re-running the backup process again.
Please contact HashiCorp Support if any additional errors occur.