How to deploy Terraform Enterprise on hardened Kubernetes environments

Introduction

When deploying Terraform Enterprise in a hardened Kubernetes environment, you may encounter startup failures. This can occur if your environment enforces strict Kubernetes Pod Security Standards, which prevent the default Terraform Enterprise deployment from running.

This guide provides the necessary configuration adjustments to the Terraform Enterprise Helm chart to successfully deploy on clusters with these security policies enabled.

Problem

With restrictive pod security standards applied, the Terraform Enterprise pod fails to start, and you may see errors similar to the following in the pod events.

Error creating: pods "terraform-enterprise-cccb7765d-fzwz2" is forbidden:
violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false
(container "terraform-enterprise" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "terraform-enterprise" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "terraform-enterprise" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "terraform-enterprise" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Prerequisites

• You are deploying Terraform Enterprise using the official Terraform Enterprise Helm chart.
• Your Kubernetes environment has restrictive pod security policies applied to the deployment namespaces.

Procedure

Follow these steps to adjust your Helm chart values and meet the security requirements of a hardened environment.

1. Apply Pod Security Labels

First, ensure your namespaces are labeled to enforce the restricted pod security standard. This is the policy that requires the subsequent configuration changes.

Execute the following commands to label the default namespaces for Terraform Enterprise and its agents.

$ kubectl label namespace terraform-enterprise pod-security.kubernetes.io/enforce=restricted
$ kubectl label namespace terraform-enterprise-agents pod-security.kubernetes.io/enforce=restricted

2. Configure the Terraform Enterprise Pod Security Context

Next, modify your values.yaml file to add the required security context for the main Terraform Enterprise container. These settings restrict privileges, drop unnecessary capabilities, and ensure the container runs as a non-root user.

## values.yaml

container:
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 1012

env:
  variables:
    TFE_RUN_PIPELINE_KUBERNETES_OPEN_SHIFT_ENABLED: "true"

The TFE_RUN_PIPELINE_KUBERNETES_OPEN_SHIFT_ENABLED environment variable is also required to enable compatibility with OpenShift-like security contexts, which is necessary for these settings to function correctly within Terraform Enterprise.

3. Create a Custom Agent Image

When a Terraform run executes, Terraform Enterprise launches an agent pod. In a hardened environment, the default agent image is not sufficient. You must build a custom agent image that is compatible with the restrictive security settings.

Follow the instructions for creating a custom agent image in the official documentation. After building and pushing the image to your container registry, proceed to the next step.

4. Configure the Agent Worker Pod Security Context

Finally, configure the agent worker pod template in your values.yaml file. Specify the custom agent image you created and apply a security context similar to the main container.

## values.yaml

agentWorkerPodTemplate:
  metadata:
    labels:
      info: pod-template-info-hashicorp
  spec:
    containers:
      - securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - ALL
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        image: <your-image-location>/custom-agent:latest
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000

Replace <your-image-location>/custom-agent:latest with the path to your custom agent image.

Outcome

After applying these settings to your Helm deployment, the Terraform Enterprise container and its agent pods will start successfully, and you will be able to execute Terraform runs in your hardened Kubernetes environment.