Introduction
In Terraform Enterprise, there is a deployment option that allows you to run a Terraform Enterprise (TFE) instance on a Nomad cluster. Additionally, you can deploy all related external services, such as PostgreSQL, Redis, and MinIO storage, on the same Nomad cluster. This article will guide you through the process of querying and connecting these external services running on Nomad cluster.
Expected Outcome
By using Nomad’s service discovery, Terraform Enterprise can dynamically discover and connect the IP addresses and ports of external services like Redis, Postgres, and MinIO running within Nomad. This eliminates the need to hardcode connection details.
Prerequisites
- You have an operational Nomad cluster (version 1.5 or newer) with native service discovery.
- The Nomad client nodes have bridge mode enabled
- You have already deployed PostgreSQL, Redis, and MinIO on the same Nomad cluster.
Use Case
A Terraform Enterprise requires all external services running on the same Nomad cluster.
Procedure
1. Query external services IPs and Ports
To enable Terraform Enterprise to connect to external services, it is essential to identify the following addresses and ports from the running external services. It is recommended to avoid hardcoding the IP addresses and ports, as these values may change over time due to Nomad client node reboots or job redeployments. Instead, you can use Nomad's service discovery to dynamically resolve the current addresses and ports of the running external services in Nomad.
TFE_REDIS_HOST
TFE_DATABASE_HOST
TFE_OBJECT_STORAGE_S3_ENDPOINT
When your external services are running on Nomad, their services information can be queried using the nomadService and nomadServices functions in Nomad’s service discovery catalog.
For example, you can define your PostgreSQL service in the Nomad Job Specification using the following code snippet:
service {
name = "postgres-svc"
port = "postgres"
provider = "nomad"
}
You can retrieve the IP address and port of service postgres-svc from Nomad's service discovery using the following template:
template {
data = <<EOF
{{ range nomadService "postgres-svc" }}
TFE_DATABASE_HOST = "{{.Address}}:{{.Port}}"
{{ end }}
EOF
env = true
destination = "secrets/ext.env"
}
In this example, Nomad will return the IP address and port from the running service postgres-svc via service discovery and assign it to the environment variable TFE_DATABASE_HOST
.
2. Example of template block in Terraform Enterprise JobSpec
Next, you’ll want to query all three external services minio-svc, redis-svc, and postgres-svc that are already registered in the Nomad service catalog. The template should look like this:
...
task "tfe-task" {
driver = "docker"
...
template {
data = <<EOF
{{ range nomadService "redis-svc" }}
TFE_REDIS_HOST = "{{.Address}}:{{.Port}}"
{{ end }}
{{ range nomadService "postgres-svc" }}
TFE_DATABASE_HOST = "{{.Address}}:{{.Port}}"
{{ end }}
{{ range nomadService "minio-svc" }}
TFE_OBJECT_STORAGE_S3_ENDPOINT = "http://{{.Address}}:{{.Port}}"
{{ end }}
EOF
env = true
destination = "secrets/ext.env"
}
}
...
Using this template example, the values of TFE_REDIS_HOST
, TFE_DATABASE_HOST
and TFE_OBJECT_STORAGE_S3_ENDPOINT
can be dynamically set as environment variables without needing to hardcode them:
You can incorporate this template into your Terraform Enterprise Job Spec under the task block before deploying Terraform Enterprise on Nomad.
3. Verify the service discovery
If you want to verify the current IP addresses and Ports of the running services, you can run Nomad service command to retrieve the data. For example:
$ nomad service info -json minio-svc