Terraform Enterprise UI Returns "Error Fetching Data: Internal Server Error"
Problem
The Terraform Enterprise web UI occasionally displays the error message 'error fetching data internal server error' when you attempt to access workspaces. Terraform runs are not affected by this UI error.
Cause
This issue commonly occurs due to connectivity disruptions with the external PostgreSQL database. The atlas.log may show errors related to the database connection, such as the database having too many clients.
error=ActiveRecord::ConnectionNotEstablished id=xxxxx message=connection to server at "10.x.x.x", port 5432 failed: FATAL: sorry, too many clients already
This initial error is often followed by multiple failed GET requests for workspaces, resulting in a 500 status code.
"method":"GET","path":"/api/v2/workspaces/ws-xxxxxx","format":"jsonapi","status":500
The log entries indicate that the PostgreSQL instance has exceeded its max_connections limit, which prevents Terraform Enterprise from retrieving workspace information for the UI.
Solutions
To resolve this issue, review the performance of your PostgreSQL instance and take the following actions.
Solution 1: Check Connection Usage
Check the current connection usage to identify if too many active or idle connections are consuming resources. You can query the pg_stat_activity view to inspect current connections.
Solution 2: Increase the Connection Limit
Increase the max_connections limit in your PostgreSQL configuration if the instance and its underlying hardware can support a higher value. Alternatively, review and optimize existing application connections to reduce the overall count.
Solution 3: Allocate Sufficient Memory
Each connection in PostgreSQL consumes memory. Ensure your server has enough capacity to support the required number of connections and allocate additional memory as needed.
Outcome
After implementing these changes, the Terraform Enterprise web UI should operate correctly without displaying connection-related error messages.