How to configure Terraform Enterprise on AKS with workload identity – HashiCorp Help Center

Introduction

With the release of Terraform Enterprise v202410-1 and later, you can use workload identity on Azure Kubernetes Service (AKS) to authenticate to the Azure storage account.

Expected Outcome

Terraform Enterprise will successfully use an AKS workload identity to connect to its object storage backend.

Prerequisites

Terraform Enterprise version v202410-1 or newer.
An active Azure Kubernetes Service (AKS) cluster.

Procedure

Ensure the AKS environment is configured to support workload identity. Refer to the official Azure workload identity documentation for guidance.
If you use Terraform to create the Kubernetes environment, set the following parameters to true in your azurerm_kubernetes_cluster resource.
```
resource "azurerm_kubernetes_cluster" "example" {
  # ...

  oidc_issuer_enabled     = true
  workload_identity_enabled = true
}
```
Verify that your managed identity has the correct roles assigned to connect to the storage account. Required roles include Storage Blob Data Owner and Storage Blob Data Contributor.
Confirm that the federated credentials have the correct subject identifier, following the format system:serviceaccount:<namespace>:<service_account_name>.

For example: system:serviceaccount:terraform-enterprise:terraform-enterprise

Add the following configuration to your overrides.yaml file, as described in the Using AKS with workload identity documentation. Replace the client ID placeholder with the client ID from your managed identity.

serviceAccount:
  enabled: true
  name: "terraform-enterprise"
  annotations:
    azure.workload.identity/client-id: <client_id_from_managed_identities>
  labels:
    azure.workload.identity/use: "true"
pod:
  serviceAccountName: "terraform-enterprise"
  labels:
    azure.workload.identity/use: "true"
agents:
  rbac:
    enabled: true

Configure the object storage settings in your Terraform Enterprise configuration.

Note: In Terraform Enterprise v202410-1, a known issue requires you to set TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY with a non-empty, base64-encoded dummy value for the configuration to be accepted. Additionally, TFE_OBJECT_STORAGE_AZURE_USE_MSI must be set to false when using workload identity.
```
## Object storage settings.
TFE_OBJECT_STORAGE_TYPE: "azure"
TFE_OBJECT_STORAGE_AZURE_ACCOUNT_NAME: exampleaccount
TFE_OBJECT_STORAGE_AZURE_CONTAINER: examplecontainer
TFE_OBJECT_STORAGE_AZURE_USE_MSI: false
## Add the dummy value for v202410-1
TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY: bmljb2tub3dzYmVzdAo=
```

Troubleshooting

If the Terraform Enterprise container fails to start due to an incorrect workload identity setup, you may see authentication errors in the logs.

{"component":"terraform-enterprise","log":"2024-10-28T15:28:37.452Z [ERROR] terraform-enterprise: check failed: name=cloud-sdk duration=47.198819ms"}
{"component":"terraform-enterprise","log":" err=\""}
{"component":"terraform-enterprise","log":" | failed to upload test file to Azure Blob Storage: PUT https://tfe17patrick.blob.core.windows.net/tfe17-container/tfe17-container/startupcheck"}
{"component":"terraform-enterprise","log":" | --------------------------------------------------------------------------------"}
{"component":"terraform-enterprise","log":" | RESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature."}
{"component":"terraform-enterprise","log":" | ERROR CODE: AuthenticationFailed"}
{"component":"terraform-enterprise","log":" | --------------------------------------------------------------------------------"}
{"component":"terraform-enterprise","log":" | \\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature."}

To verify your configuration, you can use the Azure CLI from within a test pod in your cluster.

Create a test pod using the following manifest. Update the namespace, client-id, and serviceAccount to match your environment.

apiVersion: v1
kind: Pod
metadata:
  name: az-test-pod
  namespace: terraform-enterprise # Match to your environment
  labels:
    run: az
    azure.workload.identity/use: "true"
  annotations:
    azure.workload.identity/client-id: 19c64dec-9824- # Match to your environment
spec:
  serviceAccount: terraform-enterprise # Match to your environment
  containers:
  - image: mcr.microsoft.com/azure-cli:cbl-mariner2.0
    name: az
    command: ["sleep", "3000"]
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always

Connect to the running pod.

$ kubectl -n terraform-enterprise exec -it az-test-pod -- bash

Verify that the workload identity credentials are set as environment variables.

$ env | grep AZURE

The output should be similar to the following.

AZURE_TENANT_ID=237fbc04-c52a-458b-af97-eaf7157c0cd4
AZURE_FEDERATED_TOKEN_FILE=/var/run/secrets/azure/tokens/azure-identity-token
AZURE_AUTHORITY_HOST=https://login.microsoftonline.com/
AZURE_CLIENT_ID=470c6940-08e8-4d43-8213-4b95e04bdf09

Test the login using the federated token. This command should succeed.

$ az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID

List the blobs in your storage container to verify read access.

$ az storage blob list --account-name tfe17patrick --container-name tfe17-container --auth-mode login --query "[].name"

Upload a test file to the container to verify write permissions.

$ az storage blob upload --account-name tfe17patrick --container-name tfe17-container --name EULA-Container.txt --file EULA-Container.txt --auth-mode login

If any of the above steps fail, there is a configuration issue in your Azure or Kubernetes environment that is outside the scope of Terraform Enterprise.

Additional Information

For more details, refer to the official Terraform Enterprise documentation on Using AKS with workload identity.