Troubleshooting Connectivity to External Services in Terraform Enterprise Flexible Deployment Options (Kubernetes)

Avatar
Nathan P.
  • Updated

Introduction

Terraform Enterprise flexible deployments on Kubernetes operate in an Active/Active mode, which requires external PostgreSQL, Redis, and object storage services. During new or existing installations, you may need to troubleshoot connectivity to these external services to identify the cause of startup failures or degraded performance.

This guide provides several methods for testing connectivity using standard command-line utilities.

Troubleshooting Methods

This article outlines four primary methods for testing connectivity to external services.

Method 1: Connecting from the Terraform Enterprise Container

If the Terraform Enterprise pod is running and not in a CrashLoopBackoff status, you can connect to external services directly from the container. This method is useful when you need to query the database or interact with Redis directly.

The following commands source files generated at runtime that contain the rendered connection URLs as shell variables.

Note: The environment variables ($DATABASE_URL, $REDIS_URL) in the commands below are set within the container's shell session and should not be manually interpolated.

PostgreSQL

Connect to the external PostgreSQL instance from the Terraform Enterprise container.

  • Terraform Enterprise v202404-1 and later
$ kubectl exec -ti -n <TFE_NAMESPACE> <TFE_POD> -- /bin/bash -c '. run/terraform-enterprise/atlas/atlas-env && psql $DATABASE_URL'
  • Terraform Enterprise earlier than v202404-1
$ kubectl exec -ti -n <TFE_NAMESPACE> <TFE_POD> -- /bin/bash -c '. atlas-env && psql $DATABASE_URL'

Redis

Connect to the external Redis instance from the Terraform Enterprise container.

  • Terraform Enterprise v202404-1 and later
$ kubectl exec -ti -n <TFE_NAMESPACE> <TFE_POD> -- /bin/bash -c '. run/terraform-enterprise/atlas/atlas-env && redis-cli -u $REDIS_URL'
  • Terraform Enterprise earlier than v202404-1
$ kubectl exec -ti -n <TFE_NAMESPACE> <TFE_POD> -- /bin/bash -c '. atlas-env && redis-cli -u $REDIS_URL'

Method 2: Connecting with a Debug Pod

Connectivity issues often cause startup failures, putting the Terraform Enterprise pod into a restart loop (CrashLoopBackoff). In these cases, you can use a debug pod to test connectivity.

You can create a debug pod in one of two ways.

  1. Create a copy of the Terraform Enterprise pod, which allows for interactive execution of command-line utilities included in the hashicorp/terraform-enterprise image.

    $ kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --copy-to=tfe-debugger --container=terraform-enterprise -- /bin/bash

  2. Alternatively, create a debug container within the existing Terraform Enterprise pod that uses the same container image.

    $ kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --image=<TFE_IMAGE> -- /bin/bash

Both approaches start an interactive shell session from which you can run the following commands.

PostgreSQL

Connect to the external PostgreSQL instance using psql and enter the password when prompted.

$ psql "<TFE_DATABASE_PARAMETERS> dbname=<TFE_DATABASE_NAME>" -h <TFE_DATABASE_HOST> -p <DATABASE_PORT> -U <TFE_DATABASE_USER>

Redis

Connect to the external Redis server using redis-cli and enter the password when prompted.

$ redis-cli -h <TFE_REDIS_HOSTNAME> -p <REDIS_PORT> [--tls] [--user] [--askpass]

Method 3: Testing Object Storage Connectivity

Cloud provider command-line utilities like the AWS CLI, Azure CLI, and gcloud CLI are not included in the hashicorp/terraform-enterprise image. To test object storage connections, create a debug container in the Terraform Enterprise pod using a public image from the respective cloud provider.

AWS S3

  1. Start a debug container using the amazon/aws-cli image.

    $ kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --image=amazon/aws-cli -- /bin/bash

  2. Configure authentication with the access keys used in your Terraform Enterprise configuration.

    $ aws configure

  3. Confirm the authenticated entity, for example, to verify instance profile authentication.

    $ aws sts get-caller-identity

  4. List objects in the S3 bucket.

    $ aws s3 ls s3://<TFE_BUCKET>

Azure Blob Storage

  1. Start a debug container using the mcr.microsoft.com/azure-cli image.

    $ kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --image=mcr.microsoft.com/azure-cli:cbl-mariner2.0 -- /bin/bash

  2. Log in using a managed identity if applicable.

    $ az login --identity [--username <TFE_OBJECT_STORAGE_AZURE_CLIENT_ID>]

  3. Confirm the authenticated identity.

    $ az account show

  4. List blobs in the storage container. Provide the storage account key if applicable.

    $ az storage blob list --container-name <TFE_OBJECT_STORAGE_AZURE_CONTAINER> --account-name <TFE_OBJECT_STORAGE_AZURE_ACCOUNT_NAME> [--account-key <TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY>]

Google Cloud Storage

  1. Start a debug container using the google/cloud-sdk image.

    $ kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --image=google/cloud-sdk -- /bin/bash

  2. Authenticate using a JSON key file if you are not using an attached service account.

    $ gcloud auth activate-service-account --key-file=<(echo <TFE_OBJECT_STORAGE_GOOGLE_CREDENTIALS>)

  3. List objects in the storage bucket.

    $ gsutil ls gs://<TFE_OBJECT_STORAGE_GOOGLE_BUCKET>

Method 4: Testing General TCP Connectivity and DNS

You can use common networking utilities to test basic connectivity and DNS resolution.

TCP Connectivity with curl

The curl utility is available in the hashicorp/terraform-enterprise image and can test a plain TCP connection to a given host and port.

$ curl -v --connect-timeout 2 telnet://<HOSTNAME>:<PORT>

DNS and TCP Connectivity with busybox

For tools like nslookup and netcat, start a debug container in the Terraform Enterprise pod using the busybox image.

$ kubectl debug -n <TFE_NAMESPACE> <TFE_POD> -it --image=busybox

  1. Test DNS resolution.

    $ nslookup <HOSTNAME>

  2. Test TCP connectivity.

    $ nc -vz <HOSTNAME> <PORT>

Additional Information

Articles in this section

See more

Related articles