Troubleshooting Terraform Enterprise Application Not Running After System Update – HashiCorp Help Center

Introduction

Problem

After performing a system/kernel patching update, the Terraform Enterprise (TFE) application deployed using Replicated is not running. This issue may stem from various configuration or service-related problems.

Prerequisites:

Access to the Terraform Enterprise host.
Sufficient permissions to execute system commands and modify configurations.
Basic understanding of Docker and systemd services.

Cause

The issue may arise from several potential causes, including:

Disabled IPv4 forwarding.
Docker services not running or misconfigured.
Incorrect user permissions for the Replicated user.
Firewall rules blocking necessary traffic.
Manual edits to service files leading to version mismatches.
Conflicting software or dependencies after updates.

Overview of possible solutions (if applicable)

Solutions:

1. Check IPv4 Forwarding:
Run the command:

sysctl net.ipv4.ip_forward

If it returns 0, enable it by adding net.ipv4.ip_forward = 1 to /etc/sysctl.conf, then apply changes: sysctl -p

systemctl restart network

2. Verify Docker Status: Check if Docker is running properly:

docker info
sudo systemctl daemon-reload
sudo service docker restart
systemctl status docker
docker system df
df -h
docker ps

3. Collect logs for further analysis:

journalctl --xu docker --since "YYYY-MMM-DD hh:mm:ss" > docker_05_03_2025.log 
journalctl -u replicated.service --since "YYYY-MMM-DD hh:mm:ss" > replicated_05_03_2025.log

4. Check User and Group Permissions:

Verify the replicated user:
```
cat /etc/passwd | grep replicated
```
Ensure Docker socket permissions are correct:
```
ls -la /var/run/docker.sock
```
If the replicated user is not in the Docker group, add it:
```
sudo usermod -aG docker replicated
```

Verify the change:

groups replicated  # Expected output: replicated : docker

5. Check Firewall Rules:

List current iptables rules:
```
iptables -L -v -n
```

Ensure Docker traffic is allowed:

iptables -P FORWARD ACCEPT
iptables -A INPUT -i docker0 -j ACCEPT

6. Inspect System Logs:

Check for any relevant logs in /var/log/syslog or /var/log/messages that might indicate issues after the kernel update:
```
tail -n 100 /var/log/syslog | grep "error"
tail -n 100 /var/log/messages | grep "error
```

7. Check Resource Availability:

- Ensure that there are sufficient resources (CPU, Memory, Disk Space) available on the host machine:
```
free -h
df -h
top
```

8. Validate Network Configuration:

- Ensure that network interfaces are configured correctly and are up:
```
ip addr show
```

Outcome

Following these steps should help identify and resolve issues preventing the Terraform Enterprise application from starting. Successful execution of commands will lead to a running application, allowing users to perform Terraform operations as intended.