Problem
Terraform Enterprise utilities an internal instance of Vault to encrypt and decrypt sensitive values. If there are interruptions to communication with its storage backend, it may be unable to perform these operations and Terraform Enterprise will present the following error message:
Error fetching data
Vault, a service used to store secrets in Terraform
Enterprise, has encountered an error retrieving a secret.
This can mean the vault is sealed and needs to be
unsealed to resume operations.
The ptfe_vault
(for Terraform Enterprise v202205-1 and later, the container is called tfe-vault
) container logs will contain repeated logs similar to:
2021-10-13T21:57:20.357180879Z 2021-10-13T21:57:20.357Z [ERROR] core: writing request counters to barrier: err="failed to save request counters: write tcp tfe_host:44710->postgres_host:5432: write: connection reset by peer"
Prerequisites
- Terraform Enterprise
- Internally managed Vault
- External services, utilizing an Azure managed PostgreSQL instance*
* this issue has currently been observed in Azure environments but may occur elsewhere
Cause
In some scenarios connections from Vault to the PostgreSQL instance may be terminated without sending a final FIN
message. The closed connection is returned to the connection pool instead of being evicted and is re-used for new requests causing them to fail as the connection is no longer valid and will be reset by the next hop.
This has been observed in Azure environments when restarting the PostgreSQL instance, or during failover events.
Overview of possible solutions
Solutions:
- Update to v202110-1 or later
-
Restarting the application
- Restarting the affected container,
ptfe_vault
. For Terraform Enterprise v202205-1 and later, the container is calledtfe-vault
-
Disable SSL by adding the following value to the postgres optional extra parameters in the replicated dashboard:
sslmode=disable
Outcome
If the issue persists after upgrading your Terraform Enterprise instance, please reach out to HashiCorp support for additional assistance.