Replicated Failed to Connect to Retrace Displaying this Error "ERRO [replicated-operator] operator.go:95 dial tcp <$IP-address>:9879: connect: connection refused"

Problem

During a Terraform Enterprise installation or upgrade, Replicated may fail to connect to the Retrace service. The Replicated logs show a connection refused error.

INFO [replicated-operator] eventstream.go:24 Connecting to docker stream...
ERRO [replicated-operator] operator.go:95 dial tcp <$IP-address>:9879: connect: connection refused
INFO [replicated-operator] operator.go:96 Retrying connection to replicated service...

The output of systemctl status replicated may also contain related warnings about the Retraced auditor failing to report events.

WARN models/audit/audit_event.go:77 retraced auditor failed to report daemon.params.list: Post "https://<$IP-address>:9873/publisher/v1/project/c29tZSByZWFsbHkgbG9uZyBwcm9qZWN0IGlk/event": dial >
WARN models/audit/audit_event.go:77 retraced auditor failed to report daemon.params.list: Post "<$IP-address>:9873/publisher/v1/project/c29tZSByZWFsbHkgbG9uZyBwcm9qZWN0IGlk/event": dial >

As a result, replicatedctl commands may fail, and the expected Terraform Enterprise containers may not start.

Cause

This issue can occur when the Docker userland proxy is disabled on the host. The userland proxy is responsible for forwarding traffic from the host to the appropriate Docker container ports using IP tables. If this feature is disabled, Replicated cannot establish a connection with the Retrace service, causing the installation or upgrade to fail.

The userland proxy setting is typically configured in /etc/docker/daemon.json or a custom docker.service file. The problematic configuration appears as follows.

{
  "userland-proxy": false
}

Solutions

Solution 1: Enable the Docker Userland Proxy

To resolve this issue, you must enable the Docker userland proxy.

1. Open your Docker daemon configuration file (e.g., /etc/docker/daemon.json).

2. Set the userland-proxy setting to true or remove the line entirely to use the default value (which is true).

   {
     "userland-proxy": true
   }

3. Save the configuration file.

4. Restart the Docker service to apply the changes.

   $ sudo systemctl restart docker

Outcome

After you restart Docker with the corrected configuration, Replicated should initialize successfully and connect to the Retrace service. The Terraform Enterprise containers will begin to load, and replicatedctl commands will function as expected.