Verify connectivity to PostgreSQL, Redis and S3 bucket for TFE FDO - Kubernetes installation

Introduction

Problem

Terraform Enterprise FDO on Kubernetes requires a PostgreSQL, Redis and S3 bucket to properly function and start. When one of these are not reachable the container will exit. The logs might show you the errors, but you don't always have the time to fully test what could be the issue on the pod itself. 

Example errors from the pod:

# S3 bucket
{"component":"terraform-enterprise","log":"2024-06-25T09:44:28.282Z 
[ERROR] terraform-enterprise: startup: error=\"failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, 
https response error StatusCode: 301, RequestID: 9HZJNGGAF12GNK53, HostID: 0EhgDLGAIeV1Cs6VwsWCohCiPRhcMFlACVS1VH7ertWu8UD1oh0=, 
api error PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.\""}

# PostgreSQL
No errors just a stopped pod

# Redis
No errors just a stopped pod

Prerequisites 

• Get the helm values of the Terraform Enterprise deployment:

 helm -n <namespace> get values terraform-enterprise

Solutions:

Test each with a different pod that gives you the ability to test the connection and troubleshoot.

S3 bucket on AWS

# when using the parameter TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: true
kubectl -n terraform-enterprise run s3test --rm --restart=Never --image=amazon/aws-cli -i -- s3 ls <bucket_name>

# when using the parameters TFE_OBJECT_STORAGE_S3_ACCESS_KEY_ID and TFE_OBJECT_STORAGE_S3_SECRET_ACCESS_KEY
kubectl -n terraform-enterprise run s3test --rm --restart=Never --env=AWS_ACCESS_KEY_ID=<value> --env=AWS_SECRET_ACCESS_KEY=<value> --image=amazon/aws-cli -i -- s3 ls <bucket_name>

# successful
 PRE archivistterraform/

# failure
An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

S3 bucket on Azure

# when using the parameters TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY
kubectl -n terraform-enterprise run az-test --rm --restart=Never --image=mcr.microsoft.com/azure-cli:cbl-mariner2.0 -i -- az storage blob list --account-name <TFE_OBJECT_STORAGE_AZURE_ACCOUNT_NAME>  --container-name <TFE_OBJECT_STORAGE_AZURE_CONTAINER> --account-key "<TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY>" --query "[].name"

# successful you should see files
[
"archivistterraform/json-plans/71d4662d/plan-tyxDE7QybJV62ePH",
"archivistterraform/json-provider-schemas/5a9ab298/plan-tyxDE7QybJV62ePH",
"archivistterraform/logs/apply-izPpEd9Kc6irViPY",

# failure for example
ErrorCode:ContainerNotFound

PostgreSQL

kubectl -n terraform-enterprise run psqltest --rm --env=PGPASSWORD='<password_postgres>' --restart=Never --image=postgres -i -- psql -h <postgresql_hostname> -U <username> -d <database_name> -c SELECT;

# successful
--
(1 row)

# failure
psql: error: could not translate host name "***************" to address: Name or service not known 

Redis

kubectl -n terraform-enterprise run redistest --rm --restart=Never --image=redis -i -- redis-cli -h <redis_hostname> -p <port> ping

# successful
PONG

# failure
Could not connect to Redis at **********************: Name or service not known

Outcome

You have valid options of testing the connections required by Terraform Enterprise.