How to run Terraform Enterprise FDO on EKS (AWS Kubernetes) with a specific IAM role

Introduction

Solutions

Prerequisites

• Most importantly is to make sure you follow the prerequisites from AWS for the role and permissions as outlined in this document here
• Use the Terraform Enterprise helm release >=1.3.3

Use Case

If the prerequisites have been followed you should have the following

• Kubernetes environment has an OpenIdentity provider
• A role has been created

Example of the Trust relationships:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::xxxxxxx:oidc-provider/oidc.eks.eu-north-1.amazonaws.com/id/D0D6351E1E963FD4D1D1367E51A57EEE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
            "oidc.eks.eu-north-1.amazonaws.com/id/D0D6351E1E963FD4D1D1367E51A57EEE:sub": "system:serviceaccount:terraform-enterprise:terraform-enterprise",
            "oidc.eks.eu-north-1.amazonaws.com/id/D0D6351E1E963FD4D1D1367E51A57EEE:aud": "sts.amazonaws.com"
          }
       }
     }
   ]
}

• The role has the correct policies to access the bucket

Configuration on the overrides.yamlfile used with the helm chart for Terraform Enterprise

• Alter the serviceaccount which matches the Trust relationship used on the role

serviceAccount:
  enabled: true
  name: "terraform-enterprise"
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxx:role/tfe39-storage
pod:
  serviceAccountName: "terraform-enterprise"
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxx:role/tfe39-storage

• Make sure sure the storage value match for your TFE environment

TFE_OBJECT_STORAGE_TYPE: s3
TFE_OBJECT_STORAGE_S3_BUCKET: tfe19-bucket
TFE_OBJECT_STORAGE_S3_REGION: eu-north-1
TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: true

• When Terraform Enterprise starts it should be using the role to authenticate to the storage

Troubleshooting steps

If the Terraform Enterprise container starts and the setup is incorrectly you might see errors as below

panic: error loading config: failed detecting s3 prefix: 
could not list objects: operation error S3: 
ListObjectsV2, https response error StatusCode: 403, 
RequestID: WHEAHNWXXG174DAG, 
HostID: ztdVKSJR/0FA/kdFGywAza9Dqc80fNjzTnF4Ao3KIGIis=, 
api error AccessDenied: 
User: arn:aws:sts::xxxxxxxxx:assumed-role/tfe39-storage/1730297673241545491 is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::tfe39-bucket" because no identity-based policy allows the s3:ListBucket action

Please do a test with the AWS CLI container from Amazon to verify if you are able to use the serviceaccount and reach the storage account. 

• Create a pod with the following specifications
  • Alter the namespace
  • Alter the annotation eks.amazonaws.com/role-arn
  • Alter the serviceAccount

apiVersion: v1
kind: Pod
metadata:
  name: s3test
  namespace: terraform-enterprise                                     # Match your environment
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxx:role/tfe39-storage # Match your environment
spec:
  serviceAccountName: terraform-enterprise                            # Match your environment
  containers:
  - name: aws-cli
    image: amazon/aws-cli
    command: ["sleep", "infinity"]
    restartPolicy: Never

• login to this pod

kubectl -n terraform-enterprise exec -it s3test -- bash 

• Verify if the credentials are correctly set on your pod. You should see something like the following.

bash-4.2# env | grep AWS
AWS_ROLE_ARN=arn:aws:iam::xxxxx:role/tfe39-storage
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_DEFAULT_REGION=eu-north-1
AWS_REGION=eu-north-1
AWS_STS_REGIONAL_ENDPOINTS=regional

• List the files in the container

aws s3 ls <your container>

• Upload a file to the container test the permissions

# create file
touch test.txt

# upload the file
aws s3 cp test.txt s3://<your container>/test.txt

# output
upload: ./test.txt to s3://<your container>/test.txt

• The above steps should work otherwise there is a configuration issue outside of Terraform Enterprise 

Additional Information

• AWS documentation concerning the Kubernetes configuration can be found here.