Problem
When accessing the variables page of a workspace, or querying for workspace variables through the Workspace Variables API, a 500
error is returned with the following message:
Vault, a service used to store secrets in Terraform Enterprise, has encountered an error retrieving a secret. This can mean the Vault is sealed and needs to be unsealed to resume operations.
Prerequisites
- Terraform Enterprise versions
< v202303-1 (688)
Cause
This error, which broadly indicates an issue with TFE's Vault component, can be caused by a corrupted workspace variable. This corruption is known to be caused by a bug in the Update Variables API of Terraform Enterprise versions below v202303-1 (688)
, which permits saving a literal JSON boolean when sent in the request body. This value is unable to be decrypted by Vault when accessed, as shown in the tfe-atlas
logs:
$ docker logs tfe-atlas
...
2023-06-12 19:30:32 [INFO] [active_model_serializers] Rendered ActiveModel::Serializer::CollectionSerializer with ActiveModelSerializers::Adapter::JsonApi (33.96ms)
2023-06-12 19:30:32 [ERROR] [cfd3cc13-fbd0-4111-8de6-bcdc51fb021b] exception=Vault::HTTPClientError message=The Vault server at `http://tfe-vault:8200' responded with a 400.
Any additional information the server supplied is shown below:
* invalid ciphertext: no prefix
Please refer to the documentation for help.
Solution:
To restore functionality, the corrupt variable will need to be identified and deleted from Postgres. Prior to executing the following steps, HashiCorp recommends taking a snapshot of the database as a safeguard against unexpected data loss. As the variable will be deleted and its absence potentially impact future runs on the workspace, a workspace owner should be notified about this action so they can recreate it afterward.
- Obtain the affected workspace's external ID (denoted as ID: ws-xxxxxx) from the Terraform Enterprise app by using the following trek to navigate to the workspace:
- https://$TFE-url/app/$Organization-name/workspaces/$workspace-name
- Connect to Postgres with the following command
sudo docker exec -it tfe-atlas /usr/bin/init.sh /app/scripts/wait-for-token -- bash -i -c 'psql $DATABASE_URL'
- Query for the workspace by its external ID, and note the workspace ID returned
SELECT w.id, w.name FROM rails.workspaces w WHERE w.external_id = 'WORKSPACE_EXTERNAL_ID';
- Query for all variables from the workspace, replacing
WORKSPACE_ID
with the workspace ID obtained from the previous query
SELECT v.external_id, v.key, v.value_encrypted FROM rails.vars v WHERE v.configurable_id = WORKSPACE_ID AND v.configurable_type = 'Workspace';
- Find the variable with a
value_encrypted
off
in the rows returned
external_id | key | value_encrypted
----------------------+---------------+-------------------------------------------------------
var-grEgqyoAXWwTbmtp | terraform_var | vault:v1:ODN1B6ovnpps/ac2DVtNh5ncj9haGQBIhCz2dzbjBKQZ
var-Rb7uLCcVtPbVQRYu | corrupt_var | f
(2 rows)
- Delete the variable, replacing
VARIABLE_EXTERNAL_ID
with the variable's external ID
DELETE FROM rails.vars v WHERE v.external_id = 'VARIABLE_EXTERNAL_ID';
- Exit
psql
:
\q
- Return to the variables page of the affected workspace to confirm the error is gone
Additional Information
If you continue to experience issues, please contact HashiCorp Support.