Introduction
Terraform Enterprise typically has new major version releases on a monthly schedule with new feature sets and bug fixes included. Between major releases, there are occasionally patch releases made available through the same release channel. Release notes are publicly available in the terraform-enterprise-release-notes repository, and can alternatively be found in the installer dashboard at port 8800 of the installation. HashiCorp Support encourages customers to review release notes and make sure all points are understood prior to the upgrade, or to make contact with HashiCorp Support for any inquiries.
Pre-requisites
Product Information
-
Export Terraform Enterprise Configuration
Prior to the upgrade process, HashiCorp Support strongly recommend to export existing application configurations for reference and in case they are needed for an emergency situation. The steps below are required in order to export application configuration:
- Navigate into Terraform Enterprise host machine via SSH
- Execute command
replicatedctl app-config export --hidden
- Save the JSON output into a file and store in a safe location
- Ensure there is a backup of
/etc/replicated.conf
and the required TLS certificates and license file.
-
Current and target version of Terraform Enterprise
Upgrading from much older release of Terraform Enterprise than the target release may require a stepped upgrade. The below table provides a definitive list of so called “required” versions that must be installed before upgrading further. If Terraform Enterprise is currently running an older version than any of these and will be upgraded to a more recent version, you must upgrade to those releases in the table before proceeding to the target release.
Release Sequence Date Published v201804-1 251 2018-04-05T22:20:50Z v201804-2 255 2018-04-10T21:25:26Z v201804-3 259 2018-04-18T20:35:12Z v201805-1 263 2018-05-02T20:42:35Z v201809-1 291 2018-09-06T19:01:37Z v201812-2 314 2018-12-08T01:56:59Z v201902-2 343 2019-03-14T18:03:30Z v201912-1 405 2019-12-05T23:45:28Z v202103-1 519 2021-03-23T16:47:41Z
-
Installation type
-
Online
The online installation type will require outbound communication to external networks as described in the networking requirements document. We strongly encourage customers to review the access prior to upgrade as a critical task, as having issues with network communication during upgrade may result in longer outage period and potentially cause impact to the downstream services which rely on Terraform Enterprise.
-
Airgap
Airgapped installations require the installation package to be downloaded as noted in our separate Considerations for Administrators Performing Airgapped Terraform Enterprise Upgrades article.
-
-
Backup
It is imperative to backup the data of the current Terraform Enterprise installation to avoid unexpected failures which may cause system outage or, in the worst case scenario, data loss. Terraform Enterprise offers a number of options for backup and restore method depending on operation mode, however, the common method available across operation modes is Bakup/Restore API. When creating a backup via API, it is required to provide a password and this password will need to be the same when restore Terraform Enterprise from the backup therefore this information is crucial, it is also important to note that the restoration needs to be performed against the same version of Terraform Enterprise where the backup is created, so these backups may only be used in the event of a rollback. In addition to availability of the Backup/Restore API, the following options are also commonly suggested as alternative:
- For mounted disk operational mode, a backup can be made of the disk mount location.
- For external services operation mode, backup procedures can be performed against those external services directly. The exact procedures may differ depending on the cloud provider used, or how external services are setup (for example, when using on-premise external services).
Upgrade
The upgrade procedures of Terraform Enterprise are available on this public document
In the scenario where Terraform Enterprise has been installed with pinned version, the Dashboard on the administration console port 8800 should display all the higher versions with Disabled
status due to current version is pinned. You are required to remove the pinned version with these steps prior to upgrade:
-
Connect to the Terraform Enterprise host using SSH.
-
Execute the following command to remove the pinned version. If it is more desirable to update the pinned version, the release sequence may be used in place of
0
.$ replicatedctl params set ReleaseSequence --value 0
Troubleshooting Failures On Upgrade
Replicated UI display failure with the error message Container ptfe_base_startup failed: Timeout waiting for event startup sequencing - phase 1
During the upgrade, the status of the application on the Replicated dashboard will transition to the starting up application stage. In the scenario where the status eventually transition to fail with the error Container ptfe_base_startup failed: Timeout waiting for event startup sequencing - phase 1
, the diagnosis steps below are recommended in order to proceed to the solution:
Symptom Diagnosis
- The existing installation is operating on mounted disk mode
- Navigate into Terraform Enterprise host machine via SSH
- Execute command
sudo docker logs ptfe_base_startup
and review the output if matches with the output below
2020/11/13 13:23:38 version: 91a42b2
2020/11/13 13:23:38 setting default postgres connect_timeout to 30
2020/11/13 13:23:39 amqp://rabbitmq.ptfe:5672: not ready yet: Exception (501) Reason: "read tcp 172.17.0.18:53612->172.17.0.1:5672: read: connection reset by peer"
2020/11/13 13:23:39 redis://redis:6379: ready
2020/11/13 13:23:39 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:40 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:40 amqp://rabbitmq.ptfe:5672: ready
2020/11/13 13:23:41 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:42 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:43 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:44 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:45 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:46 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
- Execute command
sudo docker logs db_pw_migrate
and review the output if matches with the output below
2020-11-13T13:23:48.505522000Z [+] executing password migration logic...
2020-11-13T13:23:48.565785000Z psql: FATAL: password authentication failed for user "hashicorp"
2020-11-13T13:23:48.566170000Z [+] updating postgres password...
2020-11-13T13:23:48.617737000Z psql: FATAL: password authentication failed for user "hashicorp"
Solution
If these indicators are all matched, the problem comes from incorrect internal PostgreSQL password being used, the How To Recover PostgreSQL Password on Mounted Disk guide is provided in order to recover the password.
Additional information
If the issue continues to occur after completing the steps in this guide, please contact HashiCorp Support to request further assistance.