Overview

This article outlines the steps to configure HashiCorp Nomad with AWS IAM Identity Provider using OpenID Connect (OIDC). This document demonstrates running a sample job where the job places a file in an S3 bucket using an assumed role and a web identity token. Additionally, operator can set up a hosted zone, generate an SSL certificate using AWS Certificate Manager (ACM), and configure an Application Load Balancer (ALB) with a DNS alias.

Prerequisite

• A running HashiCorp Nomad cluster (Nomad version 1.7.x and later).
• Nomad cluster enabled with TLS. The requirement is to use a Signed Certificate. Self Signed certificate will not work here as the AWS IAM Identity Provider can not get created with a self-sign certificate.
• An AWS account with necessary permissions to create IAM roles, policies, hosted zones, and certificates.

Steps

Step 1: Create a Hosted Zone and Generate SSL Certificate

1. Create a Hosted Zone

   • Navigate to the Route 53 console.
   • Select "Hosted zones" from the left-hand menu.
   • Click "Create hosted zone".
   • Enter the domain name (e.g., your-domain.com).
   • Note the hosted zone ID and the name servers provided.

2. Generate an SSL Certificate using ACM

   • Navigate to the ACM console.
   • Click "Request a certificate".
   • Choose "Request a public certificate".
   • Enter the domain name (e.g., your-domain.com).
   • Select "DNS validation".
   • Review and request the certificate.
   • Follow the instructions to add CNAME records to the hosted zone for validation.

Step 2: Configure Application Load Balancer (ALB) and DNS Alias

1. Create an Application Load Balancer

   • Navigate to the EC2 console.
   • Select "Load Balancers" from the left-hand menu.
   • Click "Create Load Balancer" and choose "Application Load Balancer".
   • Configure the load balancer settings (e.g., name, scheme, and IP address type).
   • Add the listeners and configure them to use the SSL certificate created in ACM.
   • Configure the target group with your Nomad server instances.
   • Complete the load balancer creation process.

2. Create a DNS Alias for ALB

   • Navigate to the Route 53 console.
   • Select the hosted zone (which was created in Step 1.1).
   • Click "Create record".
   • Choose "Alias" as the record type.
   • Select the Application Load Balancer as the alias target.
   • Complete the record creation process.

Step 3: Configure AWS IAM Identity Provider

1. Create an OIDC Identity Provider in AWS

   • Navigate to the IAM console.
   • Select "Identity providers" from the left-hand menu.
   • Click "Add provider".
   • Choose "OpenID Connect" as the provider type.
   • Enter your Nomad HTTPS URL as the OIDC provider URL (e.g. https://nomad-example.com/).
   • Enter the client ID (Audience) as "aws" that your OIDC provider will use.
   • Add the OIDC provider.

2. Create an IAM Role for OIDC Federated Users

   • Navigate to the IAM console.
   • Select "Roles" from the left-hand menu.
   • Click "Create role".
   • Select "Web identity" as the trusted entity type.
   • Choose the OIDC provider you created.
   • Select the client ID (Audience).
   • Define the role’s permissions by attaching the necessary policies (e.g., AmazonS3FullAccess for this example). User can attach as many policies as per his requirement.
   • Complete the role creation process and note the Role ARN.

Step 4: Configure Nomad for OIDC

1. Update the Nomad Configuration

   • Edit the Nomad server configuration file
   • Add the OIDC configuration (oidc_issuer) parameter as follows:

   • server {
       enabled = true
       ...
       oidc_issuer = "https://nomad-example.com/"
       ...
     }

2. Start/Restart Nomad

   • Start or restart the Nomad server agent to apply the configuration changes.

Step 5: Running a Sample Nomad Job

1. Create a Job Specification File

   • Create a file named "s3-upload.nomad.hcl" with the following specifications:

   • job "s3" {
       type = "batch"
       group "bucket" {
         task "copy" {
           driver = "docker"
           config {
             image = "public.ecr.aws/aws-cli/aws-cli"
             command = "s3"
             args = ["cp", "/local/test.txt", "s3://nomad-bucket-oidc/test-nomad.txt"]
           }
     
           identity {
             name = "aws"
             aud = ["aws"] 
             file = true
             env = true
             ttl = "1h" 
             change_mode = "restart"
           }
     
           template {
             destination = "local/test.txt"
             change_mode = "restart"
             data        = <<EOF
     Job:          {{ env "NOMAD_JOB_NAME" }}
     Alloc:        {{ env "NOMAD_ALLOC_ID" }}
     EOF
           }
     
           env {
             AWS_ROLE_ARN = "arn:aws:iam::************:role/nomad-oidc-self"
             AWS_WEB_IDENTITY_TOKEN_FILE = "${NOMAD_SECRETS_DIR}/nomad_aws.jwt" #the name format is nomad_$NAME_OF_IDENTITY.jwt
           }
     
           resources {
             cpu    = 500
             memory = 256
           }
         }
       }
     }

2. Submit The Job 

   • Submit the Job to the Nomad cluster

   • nomad job run s3-upload.nomad.hcl

3. Verify the Job Execution

   • Check the job status to ensure it is completed successfully:

   • nomad job status s3

   • Verify the file was uploaded to the S3 bucket.

Additional Points

• Nomad cluster itself can act as OIDC provider, by defining oidc_issuer parameter with nomad configuration file. Ref. https://developer.hashicorp.com/nomad/docs/configuration/server#oidc_issuer
• Need to built IAM role to built trust relationship between AWS IAM and Nomad cluster. 
• The Nomad OIDC endpoints (.well-known/openid-configuration and .well-known/jwks.json) need to be publicly reachable over HTTPS on port 443 to be verified.

Conclusion

By following these steps, user can configure HashiCorp Nomad to use AWS IAM Identity Provider with OIDC for authentication. Additionally, user has set up a hosted zone, generated an SSL certificate using ACM, and configured an Application Load Balancer with a DNS alias. Finally, user successfully ran a sample job that placed a file in an S3 bucket using an assumed role and a web identity token. This setup enhances security by leveraging AWS IAM roles and OIDC for fine-grained access control while ensuring secure access through SSL and DNS configuration.

References

Nomad Workload Identity

AWS IAM Identity Providers 

AWS Certificate Manager