How to run Terraform Enterprise FDO on EKS (AWS Kubernetes) with a specific IAM role – HashiCorp Help Center

Introduction

When running Terraform Enterprise in an AWS Elastic Kubernetes Service (EKS) cluster, you may need to assign a specific IAM role to the Terraform Enterprise pod. This allows Terraform Enterprise to use the permissions associated with that role, such as accessing its required S3 bucket for object storage.

This guide details how to configure Terraform Enterprise Flexible Deployment Options (FDO) on EKS to use an IAM role for service accounts.

Expected Outcome

After completing this guide, your Terraform Enterprise pod will be associated with a specific IAM role, enabling it to authenticate with AWS services like S3 using the permissions granted by that role.

Prerequisites

An AWS EKS cluster with an OpenID Connect (OIDC) provider configured.
An IAM role with the necessary policies to access the target S3 bucket.
The IAM role's trust relationship must be configured for the Kubernetes service account, as detailed in the AWS documentation for IAM roles for service accounts.
Terraform Enterprise helm release version 1.3.3 or newer.

Procedure

Step 1: Configure the IAM Role Trust Relationship

Ensure your IAM role's trust relationship policy allows the Kubernetes service account to assume the role. The policy should resemble the following example.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::xxxxxxx:oidc-provider/oidc.eks.eu-north-1.amazonaws.com/id/D0D6351E1E963FD4D1D1367E51A57EEE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-north-1.amazonaws.com/id/D0D6351E1E963FD4D1D1367E51A57EEE:sub": "system:serviceaccount:terraform-enterprise:terraform-enterprise",
          "oidc.eks.eu-north-1.amazonaws.com/id/D0D6351E1E963FD4D1D1367E51A57EEE:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Step 2: Configure the Terraform Enterprise Helm Chart

In the overrides.yaml file for your Terraform Enterprise Helm chart, configure the service account and pod annotations to use the IAM role ARN. The service account name must match the one specified in the IAM role's trust relationship.

serviceAccount:
  enabled: true
  name: "terraform-enterprise"
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxx:role/tfe39-storage

pod:
  serviceAccountName: "terraform-enterprise"
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxx:role/tfe39-storage

Step 3: Configure Terraform Enterprise Storage Settings

Ensure the Terraform Enterprise environment variables are configured to use S3 instance profile authentication. This allows Terraform Enterprise to inherit the IAM role from its environment.

TFE_OBJECT_STORAGE_TYPE: s3
TFE_OBJECT_STORAGE_S3_BUCKET: tfe19-bucket
TFE_OBJECT_STORAGE_S3_REGION: eu-north-1
TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: true

After applying these configurations, Terraform Enterprise should start and use the specified IAM role to authenticate to the S3 bucket.

Verification and Troubleshooting

If the Terraform Enterprise container fails to start, it may be due to incorrect permissions or configuration.

Step 1: Check for Startup Errors

An incorrect setup may produce an AccessDenied error in the container logs, similar to the following.

panic: error loading config: failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: WHEAHNWXXG174DAG, HostID: ztdVKSJR/0FA/kdFGywAza9Dqc80fNjzTnF4Ao3KIGIis=, api error AccessDenied: User: arn:aws:sts::xxxxxxxxx:assumed-role/tfe39-storage/1730297673241545491 is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::tfe39-bucket" because no identity-based policy allows the s3:ListBucket action

Step 2: Test IAM Role Association with a Debug Pod

You can use a debug pod with the AWS CLI to verify that the service account can assume the IAM role and access the S3 bucket.

Create a debug pod using the same service account and annotations. Update the namespace, role-arn, and serviceAccountName to match your environment.

apiVersion: v1
kind: Pod
metadata:
  name: s3test
  namespace: terraform-enterprise
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxx:role/tfe39-storage
spec:
  serviceAccountName: terraform-enterprise
  containers:
  - name: aws-cli
    image: amazon/aws-cli
    command: ["sleep", "infinity"]
  restartPolicy: Never

$ kubectl -n terraform-enterprise exec -it s3test -- bash

Verify that the AWS credential environment variables are set correctly within the pod.

$ env | grep AWS
## The output should be similar to the following:
AWS_ROLE_ARN=arn:aws:iam::xxxxx:role/tfe39-storage
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_DEFAULT_REGION=eu-north-1
AWS_REGION=eu-north-1
AWS_STS_REGIONAL_ENDPOINTS=regional

Test permissions by listing and uploading files to the S3 bucket.

## List objects in the bucket
$ aws s3 ls s3://<your-bucket-name>

## Create a test file
$ touch test.txt

## Upload the file to the bucket
$ aws s3 cp test.txt s3://<your-bucket-name>/test.txt
## Expected output:
## upload: ./test.txt to s3://<your-bucket-name>/test.txt

If these steps succeed, the IAM role and service account are configured correctly. If they fail, there is a configuration issue outside of Terraform Enterprise, likely with the IAM role permissions or trust policy.

Additional Information

For more details on the Kubernetes configuration, refer to the AWS documentation on IAM roles for service accounts.