Troubleshooting Terraform Enterprise Application Not Running After System Update – HashiCorp Help Center

Problem

After performing a system or kernel patch, the Terraform Enterprise application deployed with Replicated may fail to start. The application may become unresponsive or its services may not initialize correctly.

Prerequisites

You have administrative access to the Terraform Enterprise host machine.
You have permissions to execute system commands with sudo.
You have a functional understanding of Docker and systemd services.

Cause

This issue can occur due to several reasons related to the system update, including:

Disabled IPv4 forwarding, which is required for Docker networking.
The Docker service is not running or is misconfigured.
Incorrect permissions for the replicated user or the Docker socket.
New or modified firewall rules are blocking traffic required by Docker or Replicated.
Manual edits to service files that conflict with updated package versions.
Software or dependency conflicts introduced by the system update.

Solutions

Follow these solutions to diagnose and resolve the issue. Each solution addresses a potential cause.

Solution 1: Verify IPv4 Forwarding

Terraform Enterprise requires IPv4 forwarding to be enabled for container networking.

Check the current setting for IPv4 forwarding.
```
# sysctl net.ipv4.ip_forward
```
If the output is net.ipv4.ip_forward = 0, you must enable it.
To enable forwarding permanently, edit the /etc/sysctl.conf file and add or modify the following line.
```
net.ipv4.ip_forward = 1
```
Apply the changes without rebooting.
```
# sysctl -p
# systemctl restart network
```

Solution 2: Verify the Docker Service Status

Ensure the Docker daemon is running correctly and that system resources are available.

Check the overall status of Docker.
```
$ docker info
```
Reload the systemd manager configuration and restart the Docker service.
```
# systemctl daemon-reload
# service docker restart
```
Verify the Docker service is active and running.
```
# systemctl status docker
```
Check for running containers.
```
$ docker ps
```
Check disk space usage by Docker and on the host system.
```
$ docker system df
$ df -h
```

Solution 3: Inspect Firewall Rules

Firewall rules can prevent Terraform Enterprise components from communicating.

List all current iptables rules.
```
# iptables -L -v -n
```

To temporarily block outbound traffic for testing, you can add a DROP rule. This example blocks ports 80 and 443.

## Block outbound traffic on the host
# iptables -A OUTPUT -p tcp --dport 443 -j DROP
# iptables -A OUTPUT -p tcp --dport 80 -j DROP

To remove a rule, use the -D flag with the same rule specification.

# iptables -D OUTPUT -p tcp --dport 443 -j DROP
# iptables -D OUTPUT -p tcp --dport 80 -j DROP

To remove rules by number, first list them with line numbers.
```
# iptables -L OUTPUT -v -n --line-numbers
```
Then, delete the rule by its number. Note that rule numbers will shift after a deletion.
```
## This example removes rule number 1 from the OUTPUT chain
# iptables -D OUTPUT 1
```
Inspect the DOCKER-USER chain, which is the correct place to add rules for containers.
```
# iptables -L DOCKER-USER --line-numbers
```

Solution 4: Check System Resources and Permissions

Ensure the system has adequate resources and correct permissions for the application to run.

Check memory and disk space availability.
```
$ free -h
$ df -h
$ top
```
Validate the network configuration.
```
$ ip addr show
```
Verify the replicated user exists.
```
$ cat /etc/passwd | grep replicated
```
Check permissions on the Docker socket.
```
$ ls -la /var/run/docker.sock
```
If necessary, add the replicated user to the docker group.
```
# usermod -aG docker replicated
$ groups replicated
```

Solution 5: Collect and Analyze Logs

Review system and application logs for specific error messages.

Collect Docker logs from a specific timeframe.

## Replace the date and time with the relevant start time
# journalctl --xu docker --since "2023-10-27 10:00:00" > docker_logs.log

Collect Replicated service logs.

# journalctl -u replicated.service --since "2023-10-27 10:00:00" > replicated_logs.log

Check system logs for relevant errors.

# tail -n 100 /var/log/syslog | grep "error"
# tail -n 100 /var/log/messages | grep "error"

Additional Information

For more detailed information, please refer to the official Terraform Enterprise documentation, as well as the documentation for Docker and your specific Linux distribution's networking and firewall tools.