Docker Was Upgraded to 20.10.8 and Terraform Enterprise Will Not Start, Failing to Pull Docker Images

Problem

During a system patch update on the Terraform Enterprise server, Docker was upgraded to 20.10.8.  After the update, Terraform Enterprise couldn't pull docker images and the application wouldn't start.  The TFE GUI displayed the following error message on the dashboard:

"sequential image pull failed: 1 error occurred: * public/docker:latest; API error (500) Get "https:registry-1.docker.io/v2": net/http: request canceled while waiting for connection (Client Timeout exceeded while awaiting headers)"

Cause

Requests to docker.io were being blocked by the firewall, where they were previously being proxied.  There is an additional proxy entry $HTTPS_PROXY  now required within the proxy settings introduced by Docker version 20.10.8, reflected change listed in the Docker 20.10.8 release notes:

Due to net/http changes in Go 1.16, HTTP proxies configured through the $HTTP_PROXY environment variable are no longer used for TLS (https://) connections. Make sure you also set an $HTTPS_PROXY environment variable for handling requests to https:// URLs.

Solutions:

Follow the instructions on adding the required HTTPS_PROXY setting to the http-proxy.conf file.

1. Edit the http-proxy.conf file, the default location is /etc/systemd/system/docker.service.d/http-proxy.conf
2. Add HTTPS_PROXY to the http-proxy.conf file and save the file
3. Restart the docker service sudo systemctl restart docker
4. Restart Replicated either from the GUI at  https://<tfe-url>:8800/dashboard or from the commandline  replicatedctl stop, wait a minute or two and execute replicatedctl start
5. Monitor the startup progress on the TFE GUI dashboard   

Outcome

After the proxy update has been made, the TFE startup progress in the GUI should not error out at pulling docker images and move past this step completing a full Terraform Enterprise startup.  The application will be fully functional at this point.   

If issues persist after completing the steps outlined in this guide, please contact HashiCorp Support to request further assistance.

Additional Information

• Docker version 20.10.8 Release notes