Resolving HTTP 413 Errors in Terraform Enterprise on AWS EKS with NGINX Ingress Controllers – HashiCorp Help Center

Problem

When running a Terraform operation in Terraform Enterprise (TFE) on an AWS EKS environment, the run may fail with an HTTP 413 error in the logs.

2024-11-06T15:26:39.071Z [INFO] terraform: Generating and uploading provider schemas JSON
2024-11-06T15:26:45.032Z [ERROR] terraform: Unexpected HTTP response code: method=PUT url=https://tfe.yourcompany.com/_archivist/v1/object/___REDACTED___ status=413
2024-11-06T15:26:45.209Z [ERROR] terraform: Failed handling run: error="operation failed: failed uploading provider schemas JSON: unexpected status code: 413"
2024-11-06T15:26:45.607Z [INFO] agent: Shutting down

Cause

The HTTP 413 error indicates that the request payload size exceeds the server's configured limit. This typically occurs because the NGINX ingress controller in the AWS EKS environment has a default client_max_body_size limit of 1MB. Terraform operations that upload large payloads, such as provider schemas, can exceed this default limit.

Solutions

To resolve this issue, you must increase the client_max_body_size value in your NGINX ingress configuration. First, identify whether you are using the open-source ingress-nginx controller or F5's NGINX Ingress Controller, then follow the appropriate solution below.

Solution 1: Configure the ingress-nginx Controller

If you are using the open-source ingress-nginx controller, you must update the ingress resource annotations.

Locate the ingress resource manifest file that handles traffic for your Terraform Enterprise instance.
Add the nginx.ingress.kubernetes.io/proxy-body-size annotation to the manifest. This example sets the limit to 10MB, which you can adjust as needed.
```
annotations:
  nginx.ingress.kubernetes.io/proxy-body-size: "10m"
```
Apply the updated configuration to your Kubernetes cluster.
```
$ kubectl apply -f <your-ingress-resource-file.yaml>
```

Solution 2: Configure the F5 NGINX Ingress Controller

If you are using F5's NGINX Ingress Controller, you must update the ingress resource with its specific annotation.

Locate the ingress resource manifest file for your Terraform Enterprise instance.
Add the nginx.org/client-max-body-size annotation to the manifest. This example sets the limit to 10MB.
```
annotations:
  nginx.org/client-max-body-size: "10m"
```

Apply the updated configuration to your cluster.

$ kubectl apply -f <your-ingress-resource-file.yaml>

Outcome

After applying the correct annotation for your ingress controller, re-run the Terraform operation that previously failed. The operation should now complete successfully without the HTTP 413 error.

In some cases, you may need to restart the NGINX ingress controller pods for the new configuration to take effect.

$ kubectl rollout restart deployment <nginx-ingress-controller-deployment>

Additional Information