How to connect the Terraform Enterprise external services running on Nomad

Introduction

In Terraform Enterprise, there is a deployment option that allows you to run a Terraform Enterprise (TFE) instance on a Nomad cluster. Additionally, you can deploy all related external services, such as PostgreSQL, Redis, and MinIO storage, on the same Nomad cluster. This article will guide you through the process of querying and connecting these external services running on Nomad cluster.

Expected Outcome

By using Nomad’s service discovery, Terraform Enterprise can dynamically discover and connect the IP addresses and ports of external services like Redis, Postgres, and MinIO running within Nomad. This eliminates the need to hardcode connection details.

Prerequisites

• You have an operational Nomad cluster (version 1.5 or newer) with native service discovery.
• The Nomad client nodes have bridge mode enabled
• You have already deployed PostgreSQL, Redis, and MinIO on the same Nomad cluster.

Use Case

A Terraform Enterprise requires all external services running on the same Nomad cluster.

Procedure

1. Query external services IPs and Ports

To enable Terraform Enterprise to connect to external services, it is essential to identify the following addresses and ports from the running external services. It is recommended to avoid hardcoding the IP addresses and ports, as these values may change over time due to Nomad client node reboots or job redeployments. Instead, you can use Nomad's service discovery to dynamically resolve the current addresses and ports of the running external services in Nomad.

• TFE_REDIS_HOST
• TFE_DATABASE_HOST
• TFE_OBJECT_STORAGE_S3_ENDPOINT

When your external services are running on Nomad, their services information can be queried using the nomadService and nomadServices functions in Nomad’s service discovery catalog.

For example, you can define your PostgreSQL service in the Nomad Job Specification using the following code snippet:

service {
  name = "postgres-svc"
  port = "postgres"
  provider = "nomad"
}

You can retrieve the IP address and port of service postgres-svc from Nomad's service discovery using the following template:

template {
  data = <<EOF
{{ range nomadService "postgres-svc" }}
TFE_DATABASE_HOST = "{{.Address}}:{{.Port}}"
{{ end }}
EOF
  env = true
  destination = "secrets/ext.env"
}

In this example, Nomad will return the IP address and port from the running service postgres-svc via service discovery and assign it to the environment variable TFE_DATABASE_HOST.

2. Example of template block in Terraform Enterprise JobSpec

Next, you’ll want to query all three external services minio-svc, redis-svc, and postgres-svc that are already registered in the Nomad service catalog. The template should look like this:

...
    task "tfe-task" {
      driver = "docker"
      ...
      template {
        data = <<EOF
{{ range nomadService "redis-svc" }}
TFE_REDIS_HOST = "{{.Address}}:{{.Port}}"
{{ end }}
{{ range nomadService "postgres-svc" }}
TFE_DATABASE_HOST = "{{.Address}}:{{.Port}}"
{{ end }}
{{ range nomadService "minio-svc" }}
TFE_OBJECT_STORAGE_S3_ENDPOINT = "http://{{.Address}}:{{.Port}}"
{{ end }}
EOF
        env = true
        destination = "secrets/ext.env"
      }
} 
...

Using this template example, the values of TFE_REDIS_HOST, TFE_DATABASE_HOST and TFE_OBJECT_STORAGE_S3_ENDPOINT can be dynamically set as environment variables without needing to hardcode them:

You can incorporate this template into your Terraform Enterprise Job Spec under the task block before deploying Terraform Enterprise on Nomad.

3. Verify the service discovery

If you want to verify the current IP addresses and Ports of the running services, you can run Nomad service command to retrieve the data. For example:

$ nomad service info -json minio-svc

Additional Information:

• Nomad Service Discovery