Problem
When attempting to access historical data for a Terraform Enterprise workspace, such as states or run details, the operation fails with a generic error message: Internal Server Errors: Oh no, the application has errored!
Cause
Terraform Enterprise stores historical data artifacts, like state files and run logs, in an external object storage system such as AWS S3 or Azure Blob Storage. The tfe-archivist container is responsible for accessing this data. An internal server error often indicates that this container cannot communicate with the storage backend.
You can find more specific details by inspecting the container's logs on the Terraform Enterprise server instance. Run the following command to view the logs.
$ sudo docker logs tfe-archivist
An authentication failure log may appear similar to the following example.
2023/03/31 17:19:05 [ERROR] main: failed to start server { err="===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:bdbfeec3-a01e-002c-aaaa-63d6ce000000\nTime:2023-03-31T17:19:05.2065361Z, Details: \n AuthenticationErrorDetail: The MAC signature found in the HTTP request 'MeWh5xxxDokLgH+BsqkZNXWe2qb8H42uIUIETfu/U+Q=' is not the same as any computed signature. Server used following string to sign: 'GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Fri, 31 Mar 2023 17:19:02 GMT\nx-ms-version:2020-10-02\n/tfeadtstorage75aaabb7/terraform\nrestype:container'.\n" }The log output typically points to one of two primary causes:
- Invalid Credentials: The credentials configured for the object storage backend are incorrect or have expired.
- Network Failure: There is a network connectivity issue between the Terraform Enterprise instance and the object storage endpoint.
Solutions
Solution 1: Verify Object Storage Credentials
Validate the access credentials for your object storage backend outside of the Terraform Enterprise application. Use the appropriate command-line tool for your provider, such as the AWS CLI or Azure CLI, from the Terraform Enterprise host to confirm that the credentials can successfully authenticate and access the storage bucket or container.
If the credentials are invalid, update them in the Terraform Enterprise configuration and restart the application.
Solution 2: Verify Network Connectivity
Confirm that the Terraform Enterprise instance has a clear network path to the object storage endpoint. Use network diagnostic tools like traceroute or ping to test the connection from the instance to the storage provider's hostname.
Ensure that any firewalls, security groups, or network ACLs are configured to allow outbound traffic from the Terraform Enterprise instance to the object storage service on the required ports.
Outcome
After applying a solution, attempt to access the workspace data again to confirm the error is resolved. Successful access to state files and run history indicates that the connection between tfe-archivist and the object storage backend is restored.