Resolving HTTP 413 Errors in Terraform Enterprise on AWS EKS with NGINX Ingress Controllers – HashiCorp Help Center

Issue Overview

When running Terraform Enterprise (TFE) on an AWS EKS environment, you might encounter the following error during a Terraform operation:

2024-11-06T15:26:39.071Z [INFO]  terraform: Generating and uploading provider schemas JSON
2024-11-06T15:26:45.032Z [ERROR] terraform: Unexpected HTTP response code: method=PUT url=https://tfe.yourcompany.com/_archivist/v1/object/___REDACTED___ status=413
2024-11-06T15:26:45.209Z [ERROR] terraform: Failed handling run: error="operation failed: failed uploading provider schemas JSON: unexpected status code: 413"
2024-11-06T15:26:45.607Z [INFO]  agent: Shutting down

Cause of the Issue

The HTTP 413 error indicates that the request payload size exceeds the server's limit. In this specific scenario, the issue is likely caused by the client_max_body_size setting on the NGINX ingress controller in the AWS EKS environment.

By default, the client_max_body_size directive in NGINX is set to 1 MB, which can result in 413 errors when Terraform tries to upload large payloads, such as provider schemas.

Prerequisite

Identify if the NGINX ingress controller is the open-source ingress-nginx or F5's NGINX Ingress Controller and use the corresponding annotations.

Suggested Resolution

To resolve this issue, increase the client_max_body_size value in the ingress configuration. Here’s how to do it:

Steps to Fix

Locate the NGINX Ingress Resource: Identify the ingress resource handling traffic for your TFE instance. This is typically managed via an ingress controller in AWS EKS.
Update the Ingress Annotations: Add or modify the following annotation in your ingress resource to increase the payload limit:

- For the ingress-nginx controller, the annotation is nginx.ingress.kubernetes.io/client-max-body-size
- For the F5 NGINX Ingress Controller, the annotation is nginx.org/client-max-body-size
```
annotations:
  # ingress-nginx
  nginx.ingress.kubernetes.io/client-max-body-size: "10m"
  # F5 NGINX Ingress Controller
  nginx.org/client-max-body-size: "10m"
```
This example sets the limit to 10 MB, but you can adjust it further based on your requirements.
Apply the Changes: After updating the ingress resource, apply the changes to your Kubernetes cluster using kubectl:
```
kubectl apply -f <your-ingress-resource-file.yaml>
```
Restart the NGINX Pods (if necessary): In some cases, you may need to restart the NGINX ingress controller pods for the new configuration to take effect:
```
kubectl rollout restart deployment <nginx-ingress-controller-deployment>
```

Verify the Fix

After updating the ingress configuration:

Reattempt the Terraform operation that triggered the error.
Check the logs to confirm the HTTP 413 error is resolved.

Additional References