Introduction
Problem
Terraform Enterprise shows errors when deployed on Kubernetes with OIDC Authentication Integration and running the pod as NonRoot
It will show the following errors in the logs
/var/log/terraform-enterprise/backup-restore.log
2025/03/28 13:06:49 WebIdentityErr: failed fetching WebIdentity token:
caused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token
caused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied
/var/log/terraform-enterprise/archivist.log
{"@level":"error","@message":"failed retrieving key from archive store","@module":"archivist.server.http.download",
"@timestamp":"2025-03-28T13:15:38.071896Z","err":"WebIdentity
Err: failed fetching WebIdentity token: \ncaused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\n
caused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied","obj.compressed":true,"obj.encrypted":true,"obj.expire":1743257738,
"obj.key":"terraform/states/sv-1brpQLopybCavqxk/11fe66fb/72b553b0-a092-4b10-95bb-c118bd3a2d8f","obj.mode":"r","proxy":true,"req.amazon_trace_id":"-","req.id":"-"}
Terraform is not able to authenticate to AWS and have access to the S3 bucket.
Prerequisites
- Terraform Enterprise deployed on Kubernetes
- Have the OIDC Authentication Integration for Kubernetes implemented
- The pod is started as NonRoot
Cause
With the deployment of Terraform Enterprise using the helm chart you are using the following configuration variables in the overrides.yamlfile.
securityContext:
runAsUser: 1000
fsGroup: 1012
runAsNonRoot: true
env:
variables:
TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: true
The combination with Terraform Enterprise and OIDC authentication works correctly as described in this KB article here. The issue is seen seen when the securityContext is added to have the setting for runAsNonRoot
Solutions:
Option 1:
Add the openshift parameter to the configuration of you helm chart in the overrides.yaml. This adds some extra configuration to the pod environment which makes it all work as expected
securityContext:
runAsUser: 1000
fsGroup: 1012
runAsNonRoot: true
openshift:
enabled: true
env:
variables:
TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: true
Option 2:
As of now, our engineering team is actively developing the product to ensure compatibility in the next release. Once this update is released, we will update this KB article accordingly.
Outcome
Terraform Enterprise is able to start without errors in the logs.
Additional Information
-
Running Terraform Enterprise as non-root can be found here