How to configure Terraform Enterprise on AKS with workload identity – HashiCorp Help Center

Introduction

With the release of Terraform Enterprise v202410-1 you can use workload identity on AKS (Azure Kubernetes) to connect to the storage account

Expected Outcome

Terraform Enterprise uses workload identity on AKS to connect to the storage account

Prerequisites

Terraform Enterprise >= v202410-1
AKS Azure Kubernetes

Procedure

Make sure the AKS environment has been configured correctly to support workload identity. See the official Azure documentation here
When using terraform to create your Kubernetes environment then please make sure the following parameters are set to true

resource "azurerm_kubernetes_cluster" "example" {
  ...
  ...
  oidc_issuer_enabled = true
  workload_identity_enabled = true
}

Make sure that under the managed identities you have the following
- Correct roles to connect to Storage account
  - Storage Blob Data Owner
  - Storage Blob Data Contributor

Example:

- The federated credentials have the correct subject identifier
  - system:serviceaccount:<namespace_tfe>:<service_account_name>

Example: system:serviceaccount:terraform-enterprise:terraform-enterprise

Have the following configuration in your overrides.yaml file as documented here

serviceAccount:
  enabled: true
  name: "terraform-enterprise"
  annotations:
    azure.workload.identity/client-id: <client_id_from_managed_identities>
  labels:
    azure.workload.identity/use: "true"
pod:
  serviceAccountName: "terraform-enterprise"
  labels:
    azure.workload.identity/use: "true"
agents:
  rbac:
    enabled: true

In the release v202410-1 there is a small issue with the feature that requires the TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEYto be set with a non-empty base64 encoded value. (copy paste from below for example)

The TFE_OBJECT_STORAGE_AZURE_USE_MSI should be set to false.

TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY: bmljb2tub3dzYmVzdAo=
TFE_OBJECT_STORAGE_AZURE_USE_MSI: false

Troubleshooting steps

If the Terraform Enterprise container starts and the workload identity isn't setup correctly you might see errors as below

{"component":"terraform-enterprise","log":"2024-10-28T15:28:37.452Z [ERROR] terraform-enterprise: check failed: name=cloud-sdk duration=47.198819ms"}
{"component":"terraform-enterprise","log":" err="}
{"component":"terraform-enterprise","log":" | failed to upload test file to Azure Blob Storage: PUT https://tfe17patrick.blob.core.windows.net/tfe17-container/tfe17-container/startupcheck"}
{"component":"terraform-enterprise","log":" | --------------------------------------------------------------------------------"}
{"component":"terraform-enterprise","log":" | RESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature."}
{"component":"terraform-enterprise","log":" | ERROR CODE: AuthenticationFailed"}
{"component":"terraform-enterprise","log":" | --------------------------------------------------------------------------------"}
{"component":"terraform-enterprise","log":" | \\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature."}

Please do a test with the Azure CLI container from Microsoft to verify if you are able to login with workload identity and reach the storage account

Create a pod with the following specifications
- Alter the namespace
- Alter the client-id
- Alter the serviceAccount

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: az
    azure.workload.identity/use: "true"
  name: az
  namespace: terraform-enterprise                         # Match to your environment
  annotations:
    azure.workload.identity/client-id: 19c64dec-9824-     # Match to your environment
spec:
  serviceAccount: terraform-enterprise                    # Match to your environment
  containers:
  - image: mcr.microsoft.com/azure-cli:cbl-mariner2.0
    name: az
    resources: {}
    command: ["sleep", "3000"]
  dnsPolicy: ClusterFirst
  restartPolicy: Always

kubectl -n terraform-enterprise exec -it az -- bash

Verify if the workload credentials are correctly set on your pod. You should see something like the following.

root [ / ]# env | grep AZURE
AZURE_TENANT_ID=237fbc04-c52a-458b-af97-eaf7157c0cd4
AZURE_FEDERATED_TOKEN_FILE=/var/run/secrets/azure/tokens/azure-identity-token
AZURE_AUTHORITY_HOST=https://login.microsoftonline.com/
AZURE_CLIENT_ID=470c6940-08e8-4d43-8213-4b95e04bdf09

Test the login. This should succeed

az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID

List the files in the container

az storage blob list --account-name tfe17patrick --container-name tfe17-container --auth-mode login --query "[].name"

Upload a file to the container test the permissions

az storage blob upload --account-name tfe17patrick --container-name tfe17-container --name EULA-Container.txt --file EULA-Container.txt --auth-mode login

The above steps should work otherwise there is a configuration issue outside of Terraform Enterprise

Additional Information

Official documentation about workload identity on AKS can be found here