Introduction
Before deploying a Terraform Enterprise instance on Nomad, it is important to verify the connectivity and status of the external services, such as PostgreSQL, Redis, and MinIO storage, running on the same Nomad cluster. This article guides you through debugging these external services within the same Nomad cluster.
This procedure allows you to run a debugger container on Nomad and perform basic troubleshooting on external services to verify they are discoverable and reachable before deploying Terraform Enterprise.
Prerequisites
- You have an operational Nomad cluster (version 1.5 or newer) with native service discovery.
- External services (PostgreSQL, Redis, and MinIO) are running within the same Nomad cluster.
- Nomad client nodes have
bridgenetworking mode enabled.
Procedure
Step 1: Run a Debugger Job
To test connectivity to the external services, launch a lightweight job in Nomad using the following JobSpec. This Nomad job uses an appropriate/curl image and includes packages for netcat, curl, redis-cli, PostgreSQL, and the mc client. You can use this setup to execute an allocation, verify connectivity, and check the status of the external services.
This job uses Nomad's template block to dynamically resolve the IP addresses and ports of the external services and set them as environment variables inside the container.
Sample Debugger JobSpec
job "debugger" {
type = "service"
group "debug-group" {
count = 1
network {
mode = "bridge"
}
task "debug-task" {
driver = "docker"
config {
image = "appropriate/curl"
entrypoint = ["/bin/sh", "-c"]
args = [
"apk add --no-cache curl redis postgresql-client && curl -O https://dl.min.io/client/mc/release/linux-amd64/mc && chmod +x mc && mv mc /usr/local/bin/ && while true; do sleep 1000; done"
]
}
template {
data = <<EOF
{{ range nomadService "redis-svc" }}
TFE_REDIS_HOST="{{ .Address }}:{{ .Port }}"
{{ end }}
{{ range nomadService "postgres-svc" }}
TFE_DATABASE_HOST="{{ .Address }}:{{ .Port }}"
{{ end }}
{{ range nomadService "minio-svc" }}
TFE_OBJECT_STORAGE_S3_ENDPOINT="http://{{ .Address }}:{{ .Port }}"
{{ end }}
EOF
env = true
destination = "secrets/ext.env"
}
}
service {
name = "debug-service"
provider = "nomad"
}
}
}Step 2: Access the Allocation Shell
After the debugger job is deployed successfully, obtain the allocation ID by running the following command.
$ nomad job status debugger
Once you have the allocation ID, access the debugger's allocation shell using the nomad alloc exec command.
$ nomad alloc exec <ALLOC_ID> /bin/sh
Step 3: Verify Environment Variables
After you execute into the debugger's allocation, check if the environment variables are set correctly. The environment variables TFE_REDIS_HOST, TFE_DATABASE_HOST, and TFE_OBJECT_STORAGE_S3_ENDPOINT should return the current IP addresses and ports of the running external services.
/ # env | grep TFE ##... Output TFE_DATABASE_HOST=172.31.9.73:20032 TFE_REDIS_HOST=172.31.9.73:22085 TFE_OBJECT_STORAGE_S3_ENDPOINT=http://172.31.9.73:25093
Step 4: Verify MinIO Connection
Connect to MinIO and list the S3 buckets.
/ # mc alias set myminio $TFE_OBJECT_STORAGE_S3_ENDPOINT <ROOT_USER> <ROOT_PASSWORD> ##... Output Added `myminio` successfully. / # mc ls myminio ##... Output [2024-11-04 04:46:07 UTC] 0B tfebucket/
Step 5: Verify PostgreSQL Connection
Connect to PostgreSQL and list the databases.
/ # psql -h 172.31.9.73 -U postgres -p 20032 -c "\l"
##... Output
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+--------------+--------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
tfedb | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
(4 rows)Step 6: Verify Redis Connection
Connect to the Redis instance with the redis-cli client.
/ # redis-cli -h 172.31.9.73 -p 22085 ping ##... Output PONG
Outcome
Using the example debugger job and the steps outlined above, you can identify and resolve any connectivity issues with the external services. This helps ensure a successful setup of the Terraform Enterprise instance on the Nomad cluster.