Introduction
Terraform Enterprise Kubernetes deployment runs in Active/Active mode, which requires external PostgreSQL, Redis, and object storage services. In both new and existing installations, it is sometimes necessary to connect to or troubleshoot connectivity to these external services outside of Terraform Enterprise as a part of identifying the cause of startup failures or degraded performance. This guide provides several useful methods of testing connectivity under varying conditions using standard command line utilities.
Procedure
Connecting from the Terraform Enterprise Container
If the Terraform Enterprise pod is not in a CrashLoopBackoff status, connect to external services directly from the Terraform Enterprise container. This is useful when a connection to the external data storage services is required, i.e when querying the database directly. Refer to the following commands, which source files generated at runtime containing shell variables with the rendered connection URLs.
Postgres
kubectl exec -ti -n <TFE_NAMESPACE> <TFE_POD> -- /bin/bash -c '. /var/run/terraform-enterprise/atlas/atlas-env && psql $DATABASE_URL'
- Terraform Enterprise < v202404-1
kubectl exec -ti -n <TFE_NAMESPACE> <TFE_POD> -- /bin/bash -c '. atlas-env && psql $DATABASE_URL'
Redis
kubectl exec -ti -n <TFE_NAMESPACE> <TFE_POD> -- /bin/bash -c '. /var/run/terraform-enterprise/atlas/atlas-env && redis-cli -u $REDIS_URL'
- Terraform Enterprise < v202404-1
kubectl exec -ti -n <TFE_NAMESPACE> <TFE_POD> -- /bin/bash -c '. atlas-env && redis-cli -u $REDIS_URL'
The environment variable in the commands above should not be manually interpolated- it will be set in the shell session.
Connecting with Debug Pods
Connectivity issues with external services will often cause startup failures. This will make it impossible to use the commands above since Terraform Enterprise pod will enter a restart loop. Under these conditions, using debug pods can be a useful way to test connectivity. Debug pods are copies of existing pods used primarily for troubleshooting purposes.
Create a copy of the Terraform Enterprise pod, overwriting the command of the terraform-enterprise container to /bin/bash to allow for interactive execution of command line utilities included in the hashicorp/terraform-enterprise image.
kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --copy-to=tfe-debugger --container=terraform-enterprise -- /bin/bash
Alternatively, create a debug container in the existing Terraform Enterprise pod which uses the Terraform Enterprise container image.
kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --image=<TFE_IMAGE> -- /bin/bash
Either of the approaches above is acceptable. The first copies the existing pod to a new pod while the second creates a new container in the existing pod. Both will start an interactive shell session from which the following commands can be run for testing purposes.
Postgres
Connect to the external PostgreSQL instance using psql (enter password when prompted)
psql "<TFE_DATABASE_PARAMETERS> dbname=<TFE_DATABASE_NAME>" -h <TFE_DATABASE_HOST> -p <DATABASE_PORT> -U <TFE_DATABASE_USER>
Redis
Connect to the external Redis server using redis-cli (enter password when prompted)
redis-cli -h <TFE_REDIS_HOSTNAME> -p <REDIS_PORT> [--tls] [--user] [--askpass]
TCP Connectivity
The curl utility is available in the hashicorp/terraform-enterprise image and can be used to test a plain TCP connection to a given host and port.
curl -v --connect-timeout 2 telnet://<HOSTNAME>:<PORT>
Object Storage
Cloud provider command line utilities such as the AWS CLI, Azure CLI, and the gcloud CLI are not included in the hashicorp/terraform-enterprise image. To test the connection to object storage, create a debug container in the existing Terraform Enterprise pod which uses publicly available images from each respective cloud provider which have these tools.
AWS
Start a debug container in the Terraform Enterprise pod using the amazon/aws-cli image
kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --image=amazon/aws-cli -- /bin/bash
Configure authentication with access keys used in Terraform Enterprise configuration (follow prompts)
aws configure
Confirm authenticated entity (i.e to confirm instance profile authentication)
aws sts get-caller-identity
List objects in S3 bucket
aws s3 ls s3://<TFE_BUCKET>
Azure
Start a debug container in the Terraform Enterprise pod using the mcr.microsoft.com/azure-cli image
kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --image=mcr.microsoft.com/azure-cli:cbl-mariner2.0 -- /bin/bash
Log in using managed identity (optional)
az login --identity [--username <TFE_OBJECT_STORAGE_AZURE_CLIENT_ID>]
Confirm authenticated identity
az account show
List blobs in storage container (provide storage account key if applicable)
az storage blob list --container-name <TFE_OBJECT_STORAGE_AZURE_CONTAINER> --account-name <TFE_OBJECT_STORAGE_AZURE_ACCOUNT_NAME> [--account-key <TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY>]
GCP
Start a debug container in the Terraform Enterprise pod using the google/cloud-sdk image
kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --image=google/cloud-sdk -- /bin/bash
Authenticate using JSON key if not using attached service account
gcloud auth activate-service-account --key-file=<(echo <TFE_OBJECT_STORAGE_GOOGLE_CREDENTIALS>)
List objects in storage bucket
gsutil ls gs://<TFE_OBJECT_STORAGE_GOOGLE_BUCKET>
TCP Connectivity & DNS
A debug container using the busybox image can be started in the Terraform Enterprise pod to run standard network debugging utilities such as nslookup and netcat for testing DNS resolution and TCP connectivity, respectively.
kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --image=busybox
DNS
nslookup <HOSTNAME>
TCP connectivity
nc -vz <HOSTNAME> <PORT>