[TFE] Issue uploading state files larger than 512MB – HashiCorp Help Center

Introduction

Terraform Enterprise has a default limit of 512MB for state files.

Problem

After an apply in Terraform Enterprise, when a workspace has a state file larger than 512MB, the state file is not visible in the States tab of the workspace.

Manually pushing a state file bigger than 512MB also fails.

$ terraform state push ~/sv-xAY4npZExGMS0J3x.tfstate

Releasing state lock. This may take a few moments...

There was an error connecting to the remote backend. Please do not exit

Terraform to prevent data loss! Trying to restore the connection...
Still trying to restore the connection... (3s elapsed)
Still trying to restore the connection... (6s elapsed)
Still trying to restore the connection... (9s elapsed)

In the archivist.log the following error can be observed for the state version:

{"@level":"error","@message":"failed copying data to cache store", "@module":"archivist.server.http.stream-download","@timestamp":"2025-02-05T11:00:15.318439Z","cache-hit":false,
  "err":"ERR string exceeds maximum allowed size (proto-max-bulk-len)","obj.compressed":true,"obj.encrypted":true,"obj.expire":1738843213,
  "obj.key":"terraform/states/sv-xAY4npZExGMS0J3x/xxxxxxxx/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx","obj.mode":"r","obj.stream":true,"opts.framing":"*/*","opts.limit":0,"opts.offset":0,"req.amazon_trace_id":"-","req.id":"-"}

Prerequisites

Using Terraform Enterprise
Having a state file larger than 512MB

Cause

The proto-max-bulk-len attribute in the Redis config is default 512MB. If the state file is larger than 512MB it can't be cached in Redis.

Overview of possible solutions

Solutions:

1) The preferred solution is to reduce the size of your state file. This can be accomplished by reducing the number of resources in your workspace, splitting them over multiple workspaces.

2) As the issue lies within the Redis cache used by Terraform enterprise, you can also choose to run this workload locally on the CLI.

3) Adjust the proto-max-bulk-len

When using an external Redis, please have a look at your redis configuration file and see if you can adjust the proto-max-bulk-len as documented here.

# In the Redis protocol, bulk requests, that are, elements representing single
# strings, are normally limited to 512 mb. However you can change this limit
# here, but must be 1mb or greater
#
# proto-max-bulk-len 512mb

When using the internal Redis of Terraform Enterprise, there is no permanent way to set this value. A workaround is to adjust the proto-max-bulk-len via the redis cli. These commands need to be run from the terraform enterprise container.

$ redis-cli -a '<your-redis-password>'

127.0.0.1:6379> CONFIG GET proto-max-bulk-len

1) "proto-max-bulk-len"

2) "536870912"

127.0.0.1:6379> config set proto-max-bulk-len 1073741824

OK

127.0.0.1:6379> CONFIG GET proto-max-bulk-len

1) "proto-max-bulk-len"

2) "1073741824"

To find your Redis password you can use the following command:

tfectl app config --unredacted | jq .redis.password

The issue is known with engineering and they are looking into a possible solution long term.

Outcome

After adjusting the proto-max-bulk-len you will be able to run workspaces with state files larger than 512MB.

Be careful with adjusting this value to very high values, as this might have performance implications.

Additional Information