A Vault 500 Error is Returned When Accessing Workspace Variables

Problem

When you access the variables page of a workspace or query for workspace variables through the Workspace Variables API, a 500 error is returned with the following message.

Vault, a service used to store secrets in Terraform Enterprise, has encountered an error retrieving a secret. This can mean the Vault is sealed and needs to be unsealed to resume operations.

Prerequisites

• Terraform Enterprise versions < v202303-1 (688)

Cause

This error, which broadly indicates an issue with the Terraform Enterprise internal Vault component, can be caused by a corrupted workspace variable. This corruption is a known result of a bug in the Update Variables API of Terraform Enterprise versions below v202303-1 (688). The bug permits saving a literal JSON boolean value, which Vault is unable to decrypt upon access.

This is visible in the tfe-atlas container logs.

[INFO] [active_model_serializers] Rendered ActiveModel::Serializer::CollectionSerializer with ActiveModelSerializers::Adapter::JsonApi (33.96ms)
[ERROR] [cfd3cc13-fbd0-4111-8de6-bcdc51fb021b] exception=Vault::HTTPClientError message=The Vault server at `http://tfe-vault:8200' responded with a 400.Any additional information the server supplied is shown below:  * invalid ciphertext: no prefixPlease refer to the documentation for help.

Solution

To restore functionality, you must identify and delete the corrupt variable directly from the PostgreSQL database.

Note: Before you execute these steps, HashiCorp recommends taking a database snapshot to safeguard against unexpected data loss. Because the variable will be deleted, notify a workspace owner so they can recreate it after the procedure to avoid impacting future runs.

Procedure

1. Obtain the affected workspace's external ID, which is formatted as ws-xxxxxx, by navigating to the workspace URL.

   https://$TFE-url/app/$Organization-name/workspaces/$workspace-name

2. Connect to the PostgreSQL container with the following command.

   # sudo docker exec -it tfe-atlas /usr/bin/init.sh /app/scripts/wait-for-token -- bash -i -c 'psql $DATABASE_URL'

3. Query for the workspace by its external ID and note the internal workspace ID returned.

   SELECT w.id, w.name FROM rails.workspaces w WHERE w.external_id = 'WORKSPACE_EXTERNAL_ID';

4. Query for all variables from the workspace, replacing WORKSPACE_ID with the ID from the previous step.

   SELECT v.external_id, v.key, v.value_encrypted FROM rails.vars v WHERE v.configurable_id = WORKSPACE_ID AND v.configurable_type = 'Workspace';

5. Find the variable where the value_encrypted column is f.

   external_id       | key           | value_encrypted
   ----------------------+---------------+-------------------------------------------------------
   var-grEgqyoAXWwTbmtp | terraform_var | vault:v1:ODN1B6ovnpps/ac2DVtNh5ncj9haGQBIhCz2dzbjBKQZ
   var-Rb7uLCcVtPbVQRYu | corrupt_var   | f
   (2 rows)

6. Delete the corrupt variable, replacing VARIABLE_EXTERNAL_ID with its external ID.

   DELETE FROM rails.vars v WHERE v.external_id = 'VARIABLE_EXTERNAL_ID';

7. Exit the psql shell.

   \q

8. Return to the variables page of the affected workspace in the Terraform Enterprise UI to confirm the error is resolved.

Additional Information

• Workspace Variables API Documentation