TFE Task Worker Fails to Start on Docker 1.13.1 with SELinux

Problem

The Terraform Enterprise (TFE) task worker is unable to start, which causes the following degradations in functionality:

• Sentinel policy checks fail with no output.
• Terraform runs that use the agent pipeline (version 681 only) cannot be completed.
• Sentinel mocks or raw logs from a run fail to download.

The tfe-task-worker container logs a permission denied error at startup when it attempts to connect to the Docker socket.

reading config file "/ttw.hcl"
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/tfe_terraform_isolation": dial unix /var/run/docker.sock: connect: permission denied

Prerequisites

This issue affects environments with the following configuration:

• Terraform Enterprise releases 665 through 681
• Docker v1.13.1 from the Extra Packages for Enterprise Linux (EPEL) repository
• container-selinux-2.95-2.el7_6 and newer
• SELinux is enabled and in enforcing mode

Cause

This issue is caused by an SELinux constraint that blocks containers from accessing the Docker socket.

You can confirm this by checking the audit log with aureport --avc.

$ aureport --avc

##...
1301. 08/03/23 10:08:47 tfe-task-worker system_u:system_r:container_t:s0:c119,c719 42 unix_stream_socket connectto system_u:system_r:container_runtime_t:s0 denied 269476

Solution

This issue is resolved in later releases of Terraform Enterprise. For affected versions, a temporary solution is to either run SELinux in permissive mode or modify the Docker startup options to run without the --selinux-enabled flag.

Procedure

Follow these steps to modify the Docker startup option.

1. Stop the Terraform Enterprise application.

   $ replicatedctl app stop

2. Stop the Replicated services.

   $ systemctl stop replicated replicated-operator replicated-ui

3. Modify the OPTIONS variable in the /etc/sysconfig/docker file to remove the --selinux-enabled startup option.

   ## /etc/sysconfig/docker
   ## Modify these options if you want to change the way the docker daemon runs
   OPTIONS='--log-driver=journald --signature-verification=false'

4. Restart the Docker service.

   $ systemctl restart docker

5. Start the Terraform Enterprise application.

   $ replicatedctl app start