AWS Glue Streaming Job Timeout Cannot Be Reset to Unlimited in Terraform

Problem

When you manage AWS Glue Streaming Jobs with Terraform, you can configure a custom timeout. However, in older versions of the Terraform AWS Provider, once you set a timeout value, you could not revert it to the default unlimited (null) state. This caused configuration drift between your Terraform state and the actual AWS environment.

You may have observed the following behavior:

1. Creating a job without specifying a timeout results in:
   • Terraform state showing timeout = 0.
   • AWS CLI showing None (unlimited).
2. After setting a timeout, for example timeout = 500, the value updates successfully in AWS.
3. Attempting to revert the timeout to unlimited fails:

   • Setting timeout = 0 results in a Terraform error:

     Error: expected timeout to be at least (1), got 0

   • Removing the timeout argument or setting it to null produces no changes in the plan, and the AWS job does not update.

While you could clear the timeout manually from the AWS Console, Terraform and the AWS CLI did not support this action.

Prerequisites

• An AWS Glue Streaming ETL Job.
• Terraform AWS Provider versions prior to 6.20.0.
• Terraform CLI.

Cause

In versions of the Terraform AWS Provider prior to 6.20.0, the timeout argument for the aws_glue_job resource enforced a validation rule requiring a minimum value of 1. This validation prevented you from setting the value back to 0 to represent an unlimited timeout.

Solution: Upgrade the AWS Provider

This issue is resolved in version 6.20.0 of the Terraform AWS Provider. The provider now allows you to set timeout = 0 for Apache Spark Streaming ETL jobs, which correctly configures the Glue job to have an unlimited timeout.

To resolve this issue, update your configuration to use version 6.20.0 or newer of the AWS provider and set the timeout value to 0.

Example Configuration

Update your aws_glue_job resource as shown below.

resource "aws_glue_job" "streaming_job" {
  name              = "sample-streaming-job-repro"
  role_arn          = aws_iam_role.glue_job_role.arn
  glue_version      = "5.0"
  timeout           = 0 ## Allowed from provider version 6.20.0+
  number_of_workers = 2
  max_retries       = 0
  worker_type       = "G.1X"
  execution_class   = "STANDARD"
}

After applying this change, Terraform will properly update the Glue job, and the configuration will match between your Terraform state and AWS.

Additional Information

• Terraform AWS Provider: aws_glue_job resource documentation