A Vault 500 Error is Returned When Accessing Workspace Variables

Problem

When accessing the variables page of a workspace, or querying for workspace variables through the Workspace Variables API, a 500 error is returned with the following message:

Vault, a service used to store secrets in Terraform Enterprise, has encountered an error retrieving a secret. This can mean the Vault is sealed and needs to be unsealed to resume operations.

Prerequisites

• Terraform Enterprise versions < v202303-1 (688)

Cause

This error, which broadly indicates an issue with TFE's Vault component, can be caused by a corrupted workspace variable. This corruption is known to be caused by a bug in the Update Variables API of Terraform Enterprise versions below v202303-1 (688), which permits saving a literal JSON boolean when sent in the request body. This value is unable to be decrypted by Vault when accessed, as shown in the tfe-atlas logs:

$ docker logs tfe-atlas
...
2023-06-12 19:30:32 [INFO] [active_model_serializers] Rendered ActiveModel::Serializer::CollectionSerializer with ActiveModelSerializers::Adapter::JsonApi (33.96ms)
2023-06-12 19:30:32 [ERROR] [cfd3cc13-fbd0-4111-8de6-bcdc51fb021b] exception=Vault::HTTPClientError message=The Vault server at `http://tfe-vault:8200' responded with a 400.

Any additional information the server supplied is shown below:

  * invalid ciphertext: no prefix

Please refer to the documentation for help.

Solution:

To restore functionality, the corrupt variable will need to be identified and deleted from Postgres. Prior to executing the following steps, HashiCorp recommends taking a snapshot of the database as a safeguard against unexpected data loss. As the variable will be deleted and its absence potentially impact future runs on the workspace, a workspace owner should be notified about this action so they can recreate it afterward.

• Obtain the affected workspace's external ID (denoted as ID: ws-xxxxxx) from the Terraform Enterprise app by using the following trek to navigate to the workspace:
  • https://$TFE-url/app/$Organization-name/workspaces/$workspace-name
• Connect to Postgres with the following command

sudo docker exec -it tfe-atlas /usr/bin/init.sh /app/scripts/wait-for-token -- bash -i -c 'psql $DATABASE_URL'

• Query for the workspace by its external ID, and note the workspace ID returned

SELECT w.id, w.name FROM rails.workspaces w WHERE w.external_id = 'WORKSPACE_EXTERNAL_ID';

• Query for all variables from the workspace, replacing WORKSPACE_ID with the workspace ID obtained from the previous query

SELECT v.external_id, v.key, v.value_encrypted FROM rails.vars v WHERE v.configurable_id = WORKSPACE_ID AND v.configurable_type = 'Workspace';

• Find the variable with a value_encrypted of f in the rows returned

     external_id      |      key      |                value_encrypted                    
----------------------+---------------+------------------------------------------------------- 
 var-grEgqyoAXWwTbmtp | terraform_var | vault:v1:ODN1B6ovnpps/ac2DVtNh5ncj9haGQBIhCz2dzbjBKQZ
 var-Rb7uLCcVtPbVQRYu | corrupt_var   | f
(2 rows)

• Delete the variable, replacing VARIABLE_EXTERNAL_ID with the variable's external ID

DELETE FROM rails.vars v WHERE v.external_id = 'VARIABLE_EXTERNAL_ID';

• Exit psql:

\q

• Return to the variables page of the affected workspace to confirm the error is gone

Additional Information

• Workspace Variables API

If you continue to experience issues, please contact HashiCorp Support.