How to deploy Terraform Enterprise on hardened Kubernetes environments

Introduction

Customers may harden there Kubernetes environment. An example of this is to enforce pod security standards as documented here

With these settings the default Terraform Enterprise deployment example as documented here for Kubernetes will not work. Additional settings are required for Terraform Enterprise to work on these kinds of environments. 

Expected Outcome

Ability to run Terraform Enterprise on a hardened Kubernetes environment. This should prevent startup issues like the following

Note: (combined from similar events): Error creating: pods "terraform-enterprise-cccb7765d-fzwz2" is forbidden: 
violates PodSecurity "restricted:latest": allowPrivilege Escalation != false 
(container "terraform-enterprise" must set securityContext.allowPrivilegeEscalation=false), 
unrestricted capabilities (container "terraform-enterprise" must set securityContext.capabilities.drop=["ALL"]), 
runAsNonRoot != true (pod or container "terraform-enterprise" must set securityContext.runAsNonRoot=true), 
seccompProfile (pod or container "terraform-enterprise" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Prerequisites

• Deployment of Terraform Enterprise is done using the official helm chart which can be found here
• The Kubernetes environment has some hardening policies applied as documented here 

Example:

kubectl label namespace terraform-enterprise pod-security.kubernetes.io/enforce=restricted
kubectl label namespace terraform-enterprise-agents pod-security.kubernetes.io/enforce=restricted

Procedure

The Terraform Enterprise container needs to be adjusted because of these security settings with the following 

container:
 securityContext:                  
   allowPrivilegeEscalation: false 
   capabilities:                   
     drop:                         
     - ALL                         
   runAsNonRoot: true              
   seccompProfile:                 
     type: RuntimeDefault   
     
securityContext:
 runAsNonRoot: true
 runAsUser: 1000
 fsGroup: 1012
 
env:
 variables:
   TFE_RUN_PIPELINE_KUBERNETES_OPEN_SHIFT_ENABLED: "true"  

The environment variable TFE_RUN_PIPELINE_KUBERNETES_OPEN_SHIFT_ENABLED is necessary for internal Terraform Enterprise processes. This parameter will be adjusted in future versions of Terraform Enterprise. 

When a run is started on a workspace an agent pod is started to execute the code. This requires an adjusted image different from the default image as documented here

Use this altered image in your configuration with the additional security settings

agentWorkerPodTemplate:
  metadata:
    labels:
      info: pod-template-info-hashicorp
  spec:
    containers:
    - securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
          - ALL
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      image: <your-image-location>/custom-agent:latest
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000

With the above settings the Terraform Enterprise container should start successfully and your agent should be to execute Terraform runs.