TFE Task Worker Fails to Start on Docker 1.13.1 with SELinux

Problem

The TFE Task Worker is unable to start, manifesting in the following degradations in functionality:

• Sentinel Policy checks fail with no output
• Unable to complete Terraform runs using the agent pipeline (681 only)
• Failure to download Sentinel mocks or raw log from run

tfe-task-worker will log a permission denied error at startup when attempting to connect to the docker socket:

# docker logs tfe-task-worker
reading config file "/ttw.hcl"
2023/03/08 05:09:04 Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/tfe_terraform_isolation": dial unix /var/run/docker.sock: connect: permission denied

Prerequisites

• Terraform Enterprise releases 665 through 681
• Docker v1.13.1 from the Extra Packages for Enterprise Linux (EPEL) repository
• container-selinux-2.95-2.el7_6 and newer
• SELinux is enabled and enforcing

Cause

This is caused by an SELinux constraint which blocks containers from accessing the Docker socket. It can be confirmed by checking the audit log with aureport --avc:

1301. 08/03/23 10:08:47 tfe-task-worker system_u:system_r:container_t:s0:c119,c719 42 unix_stream_socket connectto system_u:system_r:container_runtime_t:s0 denied 269476

Solutions:

This has been identified as a bug and will be resolved in the next release, restoring Terraform Enterprise's compatibility with SELinux in enforcing mode. A temporary solution is to either run SELinux in permissive mode, or start Docker without the --selinux-enabled. Follow the steps below to modify the Docker startup option.

• Stop the Terraform Enterprise application:

replicatedctl app stop

• Stop Replicated services

systemctl stop replicated replicated-operator replicated-ui

• Modify the OPTIONS variable /etc/sysconfig/docker file to remove the --selinux-enabled startup option:

# /etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'

• Restart Docker

systemctl restart docker

• Start Terraform Enterprise:

replicatedctl app start

Additional Information

• If you continue to experience issues, please contact HashiCorp Support.