Problem
After an upgrade to Terraform Enterprise >=v202404-2 (mounted disk), the application fails to start. Replicated deployments display the following error in the status output.
Ready state command canceled: context deadline exceeded
The terraform-enterprise container logs contain the following permission errors which caused PostgreSQL failed to start.
tfe-1 | 2024-05-06 05:18:48,997 INFO spawned: 'postgres' with pid 63
tfe-1 | 2024-05-06 05:18:49,137 INFO exited: postgres (exit status 2; not expected)
tfe-1 | {"component":"postgres","log":"The files belonging to this database system will be owned by user \"postgres\"."}
tfe-1 | {"component":"postgres","log":"This user must also own the server process."}
tfe-1 | {"component":"postgres","log":""}
tfe-1 | {"component":"postgres","log":"The database cluster will be initialized with locale \"en_US.UTF-8\"."}
tfe-1 | {"component":"postgres","log":"The default database encoding has accordingly been set to \"UTF8\"."}
tfe-1 | {"component":"postgres","log":"The default text search configuration will be set to \"english\"."}
tfe-1 | {"component":"postgres","log":""}
tfe-1 | {"component":"postgres","log":"Data page checksums are disabled."}
tfe-1 | {"component":"postgres","log":""}
tfe-1 | {"component":"postgres","log":"initdb: error: could not access directory \"/var/lib/terraform-enterprise/postgres/pgdata\": Permission denied"}
tfe-1 | {"component":"postgres","log":"pg_ctl: could not access directory \"/var/lib/terraform-enterprise/postgres/pgdata\": Permission denied"}
tfe-1 | {"component":"postgres","log":"psql: error: connection to server on socket \"/var/run/postgresql/.s.PGSQL.5432\" failed: No such file or directory"}
tfe-1 | {"component":"postgres","log":"\tIs the server running locally and accepting connections on that socket?"}
tfe-1 | {"component":"postgres","log":"psql: error: connection to server on socket \"/var/run/postgresql/.s.PGSQL.5432\" failed: No such file or directory"}
tfe-1 | {"component":"postgres","log":"\tIs the server running locally and accepting connections on that socket?"}
tfe-1 | {"component":"postgres","log":"pg_ctl: could not access directory \"/var/lib/terraform-enterprise/postgres/pgdata\": Permission denied"}
tfe-1 | {"component":"postgres","log":"postgres: could not access directory \"/var/lib/terraform-enterprise/postgres/pgdata\": Permission denied"}
Prerequisites
- Terraform Enterprise v202404-2 or later
- Mounted disk operational mode
Cause
Terraform Enterprise v202404-2 introduced process segmentation. With this change, individual processes in the Terraform Enterprise container run under distinct user accounts whereas previously they all ran under the terraform-enterprise user. More specifically, the postgres process now runs under a dedicated postgres user.
terraform-enterprise@27a0402261d7:/$ ps -ef | grep postgres
postgres 70 57 0 15:10 ? 00:00:00 /usr/lib/postgresql/14/bin/postgres -c config_file=/etc/postgresql/postgresql.conf -c hba_file=/etc/postgresql/pg_hba.conf
terraform-enterprise@27a0402261d7:/$ id postgres
uid=103(postgres) gid=105(postgres) groups=105(postgres),104(ssl-cert)In previous installations, Postgres ran under the terraform-enterprise user, which is a member of the root group. For mounted disk installations which restrict access to the mounted data directory from other, Postgres will fail to start on this release with the errors above.
For example, assuming the disk path is /opt/hashicorp, the following permission bits on the disk path directory will no longer work in v202404-2.
# ls -ld /opt/hashicorp
drwxrwx--- 5 root root 4096 Mar 20 09:13 /opt/hashicorp
Solution
Review the permissions of the data directory on the TFE host to ensure it is executable by other.
chmod o+x /opt/hashicorp
The disk path directory is assigned root group ownership and the group permission bits are recursively copied to the user permission bits by Terraform Enterprise at startup. Additionally, <TFE_DISK_PATH>/postgres/pgdata is recursively assigned postgres:root ownership and 0700 permissions. If the disk path directory is not executable by other, however, this remains unaltered.
Additional information
If you continue to experience issues, please contact HashiCorp Support.