Resolving HTTP 413 Errors in Terraform Enterprise on AWS EKS

Issue Overview

When running Terraform Enterprise (TFE) on an AWS EKS environment, you might encounter the following error during a Terraform operation:

2024-11-06T15:26:39.071Z [INFO]  terraform: Generating and uploading provider schemas JSON
2024-11-06T15:26:45.032Z [ERROR] terraform: Unexpected HTTP response code: method=PUT url=https://tfe.yourcompany.com/_archivist/v1/object/___REDACTED___ status=413
2024-11-06T15:26:45.209Z [ERROR] terraform: Failed handling run: error="operation failed: failed uploading provider schemas JSON: unexpected status code: 413"
2024-11-06T15:26:45.607Z [INFO]  agent: Shutting down

Cause of the Issue

The HTTP 413 error indicates that the request payload size exceeds the server's limit. In this specific scenario, the issue is likely caused by the client_max_body_size setting on the NGINX ingress controller in the AWS EKS environment.

By default, the client_max_body_size directive in NGINX is set to 1 MB, which can result in 413 errors when Terraform tries to upload large payloads, such as provider schemas.

Suggested Resolution

To resolve this issue, increase the client_max_body_size value in the NGINX ingress configuration. Here’s how to do it:

Steps to Fix

1. Locate the NGINX Ingress Resource: Identify the ingress resource handling traffic for your TFE instance. This is typically managed via an ingress controller in AWS EKS.

2. Update the Ingress Annotations: Add or modify the following annotation in your ingress resource to increase the payload limit:

   annotations:
     nginx.ingress.kubernetes.io/client-max-body-size: "10m"

   This example sets the limit to 10 MB, but you can adjust it further based on your requirements.

3. Apply the Changes: After updating the ingress resource, apply the changes to your Kubernetes cluster using kubectl:

   kubectl apply -f <your-ingress-resource-file.yaml>

4. Restart the NGINX Pods (if necessary): In some cases, you may need to restart the NGINX ingress controller pods for the new configuration to take effect:

   kubectl rollout restart deployment <nginx-ingress-controller-deployment>

Verify the Fix

After updating the ingress configuration:

• Reattempt the Terraform operation that triggered the error.

• Check the logs to confirm the HTTP 413 error is resolved.

Additional References

• NGINX Ingress Controller Documentation - client_max_body_size