Terraform Enterprise Displays Vault 500 Errors, Vault Cannot Renew Tokens

Introduction

This article addresses an issue that is specific to customers who are running Terraform Enterprise in the individual installation mode that are also using an external Vault installation rather than the default internal Vault. If you are experiencing a similar issue that falls outside of these parameters, please reach out to HashiCorp support.

Problem

Occasionally, Terraform Enterprise users may see an issue where Terraform Enterprise ceases to function after a given period of time, with Vault 500 errors appearing in the UI. Upon inspecting the ptfe_vault container on the instance using docker logs ptfe_vault, administrators may find many occurrences of the following error message leading up to the failure.

[ERROR] Unable to renew vault token, retry in 60 seconds

While this error indicates that the token is not being refreshed, the token that was acquired during the start up phase of the Terraform Enterprise application will continue to function until its TTL has passed. This may delay the discovery of the issue until the time that the original token expires, and may cause additional confusion as restarting the application will allow it to return to a functional state until the initial token again expires.

Cause

In Terraform Enterprise version v201908-1, the version of the Vault CLI that is included in the ptfe_vault container was upgraded from 0.9.6 to 1.2.0. This Vault CLI version contained a deprecation of the command that Terraform Enterprise previously used to renew its Vault token.

Solution

In Terraform Enterprise version v202001-1, the command used to renew the Vault token was updated to match the new syntax appropriate for later version of Vault CLI. Upgrading to this release (or later) resolves the issue of the expiring tokens.