Error fetching data: Vault has encountered an error retrieving a secret. – HashiCorp Help Center

Problem

Terraform Enterprise utilities an internal instance of Vault to encrypt and decrypt sensitive values. If there are interruptions to communication with its storage backend, it may be unable to perform these operations and Terraform Enterprise will present the following error message:

Error fetching data
Vault, a service used to store secrets in Terraform
Enterprise, has encountered an error retrieving a secret.
This can mean the vault is sealed and needs to be
unsealed to resume operations.

The ptfe_vault (for Terraform Enterprise v202205-1 and later, the container is called tfe-vault) container logs will contain repeated logs similar to:

2021-10-13T21:57:20.357180879Z 2021-10-13T21:57:20.357Z [ERROR] core: writing request counters to barrier: err="failed to save request counters: write tcp tfe_host:44710->postgres_host:5432: write: connection reset by peer"

Prerequisites

Terraform Enterprise
Internally managed Vault
External services, utilizing an Azure managed PostgreSQL instance*

* this issue has currently been observed in Azure environments but may occur elsewhere

Cause

In some scenarios connections from Vault to the PostgreSQL instance may be terminated without sending a final FIN message. The closed connection is returned to the connection pool instead of being evicted and is re-used for new requests causing them to fail as the connection is no longer valid and will be reset by the next hop.

This has been observed in Azure environments when restarting the PostgreSQL instance, or during failover events.

Overview of possible solutions

Solutions:

Update to v202110-1 or later
Restarting the application
Restarting the affected container, ptfe_vault. For Terraform Enterprise v202205-1 and later, the container is called tfe-vault
Disable SSL by adding the following value to the postgres optional extra parameters in the replicated dashboard: sslmode=disable

Outcome

If the issue persists after upgrading your Terraform Enterprise instance, please reach out to HashiCorp support for additional assistance.

Additional Information