Introduction
Before deploying the Terraform Enterprise instance on Nomad, it’s important to verify the connectivity and status of the external services (e.g.: PostgreSQL, Redis, and MinIO storage) running on the same Nomad cluster. This article will guide you through the process of debugging these external services within the same Nomad cluster.
Prerequisites
- You have an operational Nomad cluster (version 1.5 or newer) with native service discovery.
- External services (PostgreSQL, Redis, and MinIO) are running within the same Nomad cluster.
- Nomad client nodes have bridge networking mode enabled.
Tutorial
The following steps can verify that all external services can be discovered and reachable within the Nomad cluster before deploying a Terraform Enterprise instance. You will be able to run a debugger container on the Nomad and perform basic troubleshooting on external services.
1. Run Debugger Job to Test Connectivity to External Services
To test connectivity to these external services, It's helpful to launch a lightweight job in Nomad using the following Jobspec. This Nomad job uses a appropriate/curl image and includes packages for netcat, curl, redis-cli, PostgreSQL and the mc client. You can use this setup to execute an allocation, verify connectivity, and check the status of the external services.
Sample debugger JobSpec
job "debugger" {
type = "service"
group "debug-group" {
count = 1
network {
mode = "bridge"
}
task "debug-task" {
driver = "docker"
config {
image = "appropriate/curl"
entrypoint = ["/bin/sh", "-c"]
args = [
"apk add --no-cache curl redis postgresql-client && curl -O https://dl.min.io/client/mc/release/linux-amd64/mc && chmod +x mc && mv mc /usr/local/bin/ && while true; do sleep 1000; done"
]
}
template {
data = <<EOF
{{ range nomadService "redis-svc" }}
TFE_REDIS_HOST="{{ .Address }}:{{ .Port }}"
{{ end }}
{{ range nomadService "postgres-svc" }}
TFE_DATABASE_HOST="{{ .Address }}:{{ .Port }}"
{{ end }}
{{ range nomadService "minio-svc" }}
TFE_OBJECT_STORAGE_S3_ENDPOINT="http://{{ .Address }}:{{ .Port }}"
{{ end }}
EOF
env = true
destination = "secrets/ext.env"
}
}
service {
name = "debug-service"
provider = "nomad"
}
}
}
Notice that this job uses Nomad's template to dynamically resolve the IP addresses and ports of the external services running on Nomad, and set them as environment variables inside the container.
2. Access the Allocation Shell with Nomad Exec
After the debugger job is deployed successfully, you can obtain the allocation ID by running the following command:
$ nomad job status debugger
Once you have the allocation ID, you can access the debugger's allocation shell using the nomad alloc exec command:
$ nomad alloc exec <ALLOAC_ID> /bin/sh
3. Verify Environment Variables
Once you’ve exec into the debugger’s allocation, let’s check if the environment variables are set correctly. It’s important that the environment variables TFE_REDIS_HOST, TFE_DATABASE_HOST, and TFE_OBJECT_STORAGE_S3_ENDPOINT return the current IP addresses and ports of the running external services. For example:
/ # env | grep TFE
TFE_DATABASE_HOST=172.31.9.73:20032
TFE_REDIS_HOST=172.31.9.73:22085
TFE_OBJECT_STORAGE_S3_ENDPOINT=http://172.31.9.73:25093
4. Verify MinIO Connection and List Buckets
Connect to MinIO and list S3 bucket, for example:
/ # mc alias set myminio $TFE_OBJECT_STORAGE_S3_ENDPOINT <ROOT_USER> <ROOT_PASSWORD>
Added `myminio` successfully.
/ # mc ls myminio
[2024-11-04 04:46:07 UTC] 0B tfebucket/
5. Verify PostgreSQL Connection and List Databases
Connect to PostgreSQL and list database, for example:
/ # psql -h 172.31.9.73 -U postgres -p 20032 -c "\l"
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
tfedb | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
(4 rows)
6. Verify Redis Connection Using redis-cli
Connect to Redis instance with redis-cli client
/ # redis-cli -h 172.31.9.73 -p 22085 ping
PONG
7. Final Steps: Troubleshooting Connectivity and Ensuring Successful Setup
Using the example debugger job and the steps outlined above, you can identify any connectivity issues with the external services. This will assist you in successfully setting up the Terraform Enterprise instance on the Nomad cluster.
Additional Information: