How to connect the Terraform Enterprise external services running on Nomad

Introduction

When using the Nomad deployment option for Terraform Enterprise, you can run the instance and its external services, such as PostgreSQL, Redis, and MinIO storage, on the same Nomad cluster. This article guides you through querying and connecting to these external services.

This approach is useful when a Terraform Enterprise instance requires all external services to run on the same Nomad cluster.

Expected Outcome

By using Nomad’s native service discovery, Terraform Enterprise can dynamically discover and connect to the IP addresses and ports of external services. This process eliminates the need to hardcode connection details.

Prerequisites

• An operational Nomad cluster (version 1.5 or newer) with native service discovery enabled.
• The Nomad client nodes have bridge mode enabled.
• You have deployed PostgreSQL, Redis, and MinIO on the same Nomad cluster.

Procedure

Step 1: Query external service addresses and ports

To enable Terraform Enterprise to connect to external services, you must identify the addresses and ports for the following environment variables:

• TFE_REDIS_HOST
• TFE_DATABASE_HOST
• TFE_OBJECT_STORAGE_S3_ENDPOINT

Avoid hardcoding IP addresses and ports, as these values may change when Nomad client nodes reboot or jobs redeploy. Instead, use Nomad's service discovery to dynamically resolve the current addresses and ports.

When your external services run on Nomad, you can query their service information using the nomadService and nomadServices functions in Nomad’s service discovery catalog.

For example, define your PostgreSQL service in the Nomad Job Specification with the following code.

service {
  name     = "postgres-svc"
  port     = "postgres"
  provider = "nomad"
}

You can retrieve the IP address and port of the postgres-svc service from Nomad's service discovery using the following Nomad template block.

template {
  data = <<EOF
{{ range nomadService "postgres-svc" }}
TFE_DATABASE_HOST = "{{.Address}}:{{.Port}}"
{{ end }}
EOF

  env         = true
  destination = "secrets/ext.env"
}

In this example, Nomad retrieves the IP address and port from the running postgres-svc service and assigns the combined value to the TFE_DATABASE_HOST environment variable.

Step 2: Define the template block in the Terraform Enterprise JobSpec

Next, query all three external services (minio-svc, redis-svc, and postgres-svc) that are registered in the Nomad service catalog. The template block should appear as follows.

# ...
task "tfe-task" {
  driver = "docker"
  # ...

  template {
    data = <<EOF
{{ range nomadService "redis-svc" }}
TFE_REDIS_HOST = "{{.Address}}:{{.Port}}"
{{ end }}
{{ range nomadService "postgres-svc" }}
TFE_DATABASE_HOST = "{{.Address}}:{{.Port}}"
{{ end }}
{{ range nomadService "minio-svc" }}
TFE_OBJECT_STORAGE_S3_ENDPOINT = "http://{{.Address}}:{{.Port}}"
{{ end }}
EOF

    env         = true
    destination = "secrets/ext.env"
  }

  # ...
}
# ...

Using this template, the values of TFE_REDIS_HOST, TFE_DATABASE_HOST, and TFE_OBJECT_STORAGE_S3_ENDPOINT are dynamically set as environment variables.

Incorporate this template into your Terraform Enterprise Job Specification under the task block before deploying Terraform Enterprise on Nomad.

Step 3: Verify service discovery

To verify the current IP addresses and ports of the running services, run the nomad service command to retrieve the data. For example, to check the minio-svc service, run the following command.

$ nomad service info -json minio-svc

Additional Information

• Nomad Service Discovery