Problem
When trying a backup/restore of Terraform Enterprise, backup is initiated on server A, while the restore is done on server B. The following error message may present itself.
413 Request Entity Too Large
Cause
- From the nginx logs the following error is showing:
2022-02-21T11:47:44.151469840Z 2022/02/21 11:47:44 [error] 10#10: *24 connect()
failed (111: Connection refused) while connecting to upstream,
client: 172.19.0.1, server: , request: "GET /_health_check HTTP/1.1",
upstream: "http://172.19.0.19:9292/_health_check", host: "TFE IP
2022-02-17T09:11:41.065925125Z 2022/02/17 09:11:41 [error] 9#9:
*5600 client intended to send too large body: 52735116065 bytes,
client: 10.5.110.36, server: , request: "POST /_backup/api/v1/restore HTTP/1.1",
host: "TFE_ip.eu-west-1.compute.internal"
This error message means:
-
Terraform Enterprise Restore API is failling into a different server block that doesn't have the max client body set or with a limited client max size.
Solution:
-
Make sure the request is going to the hostname configured in the application's
hostnamesetting, otherwise we won't use the correct server block. -
By doing that, the request with the hostname instead of the IP. And nginx sees that hostname
-
In
/etc/hostscheck its content if private IP is not there added
for example
# tail -n 1 /etc/hosts
10.0.xx.1xx TFE-xxx.xxx.com
- Then run command against FQDN which is showing/added in the hosts files
curl --header "Authorization: Bearer REDACTED" --request POST --form \
config=@payload.json --form snapshot=@backup.blob \
https://TFE-xxx.xxx.com/_backup/api/v1/restore
snapshot applied successfully