Replicated Failed to Connect to Retrace Displaying this Error "ERRO [replicated-operator] operator.go:95 dial tcp <$IP-address>:9879: connect: connection refused"

Problem

During an attempt to install or upgrade Terraform Enterprise, Replicated may fail to connect to Retrace exhibited by this error reported in the replicated logs: 

INFO 2024-05-08T14:09:05+00:00 [replicated-operator] eventstream.go:24 Connecting to docker stream...
ERRO 2024-05-08T14:09:05+00:00 [replicated-operator] operator.go:95 dial tcp <$IP-address>:9879: connect: connection refused
INFO 2024-05-08T14:09:05+00:00 [replicated-operator] operator.go:96 Retrying connection to replicated service...

The output systemctl status replicated contains the following error logs:

May 16 19:47:43 wh-1ec4590efe.svr.us.jpmchase.net docker[2739446]: WARN 2024-05-16T19:47:43+00:00 models/audit/audit_event.go:77 retraced auditor failed to report daemon.params.list: Post "https://<$IP-address>:9873/publisher/v1/project/c29tZSByZWFsbHkgbG9uZyBwcm9qZWN0IGlk/event": dial >
May 16 19:48:24 <$hostname> docker[2739446]: WARN 2024-05-16T19:48:24+00:00 models/audit/audit_event.go:77 retraced auditor failed to report daemon.params.list: Post "<$IP-address>:9873/publisher/v1/project/c29tZSByZWFsbHkgbG9uZyBwcm9qZWN0IGlk/event": dial >
ESCOC

Additionally, replicatedctl commands may error during processing and the expected containers may not load.

Cause

• The Docker userland proxy is not configured properly in accordance to the host network settings, using IP tables for port forwarding. This is not allowing Replicated to connect to Retrace.
• The Docker userland proxy is responsible for forwarding traffic from the host machine to the appropriate Docker container ports. The userland proxy adds an extra layer between the host and the containers, which can sometimes cause issues with port forwarding or network address translation.
• The userland proxy setting by default is located in the /etc/docker/daemon.json or possibly within a customized docker.service file that may reference a separate configuration location.  

  • DisableUserlandProxy=true

Solutions:

• Ensure the the Docker userland proxy is not disabled by changing the setting to false: DisableUserlandProxy=false

• Save the changes in the file

• Restart the Docker service with sudo systemctl restart docker

Outcome

• Replicated should complete initiatization and connect to Retraced

• The containers will begin to load and the replicatedctl commands will begin working again