Terraform Enterprise OIDC Authentication Integration with runAsNonRoot error

Problem

When Terraform Enterprise is deployed on Kubernetes with OIDC authentication and configured to run as a non-root user, the application fails to start. Terraform Enterprise cannot authenticate to AWS to access the S3 bucket for state storage.

The following permission denied errors appear in the logs.

In backup-restore.log:

WebIdentityErr: failed fetching WebIdentity token: caused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token
caused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied

In archivist.log:

{"@level":"error","@message":"failed retrieving key from archive store","@module":"archivist.server.http.download","err":"WebIdentityErr: failed fetching WebIdentity token: \ncaused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied","obj.compressed":true,"obj.encrypted":true,"obj.expire":1743257738,"obj.key":"terraform/states/sv-1brpQLopybCavqxk/11fe66fb/72b553b0-a092-4b10-95bb-c118bd3a2d8f","obj.mode":"r","proxy":true,"req.amazon_trace_id":"-","req.id":"-"}

Prerequisites

• Terraform Enterprise deployed on Kubernetes.
• OIDC authentication integration for Kubernetes is implemented.
• The Terraform Enterprise pod is configured to start as a non-root user (runAsNonRoot: true).

Cause

The issue occurs when using a securityContext in the Helm chart's overrides.yaml file to run the pod as a non-root user. The specified user does not have the necessary permissions to read the OIDC service account token required for AWS authentication.

Example problematic configuration:

securityContext:
  runAsUser: 1000
  fsGroup: 1012
  runAsNonRoot: true
env:
  variables:
    TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: true

While the combination of Terraform Enterprise and OIDC authentication is supported, as described in How to run Terraform Enterprise FDO on EKS (AWS Kubernetes) with a specific IAM role, adding the runAsNonRoot security context introduces this permission issue.

Solutions

Solution 1: Enable the OpenShift Compatibility Setting

Add the openshift.enabled: true parameter to the configuration in your overrides.yaml file. This setting applies additional security configurations to the pod environment that grant the necessary permissions for the non-root user to access the service account token.

Update your configuration as follows.

securityContext:
  runAsUser: 1000
  fsGroup: 1012
  runAsNonRoot: true
openshift:
  enabled: true
env:
  variables:
    TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: true

Outcome

After applying the configuration change and redeploying, Terraform Enterprise starts successfully without authentication errors in the logs.

Additional Information

• For more details on this configuration, refer to the documentation on Running Terraform Enterprise as a non-root user.
• The Terraform Enterprise engineering team is working to improve compatibility for this scenario in a future release.