Upgrading Terraform Enterprise

Introduction

Terraform Enterprise typically has new major version releases on a monthly schedule with new feature sets and bug fixes included. Between major releases, there are occasionally patch releases made available through the same release channel. Release notes are publicly available in the terraform-enterprise-release-notes repository, and can alternatively be found in the installer dashboard at port 8800 of the installation. HashiCorp Support encourages customers to review release notes and make sure all points are understood prior to the upgrade, or to make contact with HashiCorp Support for any inquiries.

Pre-requisites

Product Information

• Export Terraform Enterprise Configuration

  Prior to the upgrade process, HashiCorp Support strongly recommend to export existing application configurations for reference and in case they are needed for an emergency situation. The steps below are required in order to export application configuration:

  1. Navigate into Terraform Enterprise host machine via SSH
  2. Execute command replicatedctl app-config export --hidden
  3. Save the JSON output into a file and store in a safe location
  4. Ensure there is a backup of /etc/replicated.conf and the required TLS certificates and license file.

• Current and target version of Terraform Enterprise

  Upgrading from much older release of Terraform Enterprise than the target release may require a stepped upgrade. The below table provides a definitive list of so called “required” versions that must be installed before upgrading further. If Terraform Enterprise is currently running an older version than any of these and will be upgraded to a more recent version, you must upgrade to those releases in the table before proceeding to the target release.

  Release	Sequence	Date Published
  v201804-1	251	2018-04-05T22:20:50Z
  v201804-2	255	2018-04-10T21:25:26Z
  v201804-3	259	2018-04-18T20:35:12Z
  v201805-1	263	2018-05-02T20:42:35Z
  v201809-1	291	2018-09-06T19:01:37Z
  v201812-2	314	2018-12-08T01:56:59Z
  v201902-2	343	2019-03-14T18:03:30Z
  v201912-1	405	2019-12-05T23:45:28Z
  v202103-1	519	2021-03-23T16:47:41Z

• Installation type

  • Online

    The online installation type will require outbound communication to external networks as described in the networking requirements document. We strongly encourage customers to review the access prior to upgrade as a critical task, as having issues with network communication during upgrade may result in longer outage period and potentially cause impact to the downstream services which rely on Terraform Enterprise.

  • Airgap

    Airgapped installations require the installation package to be downloaded as noted in our separate Considerations for Administrators Performing Airgapped Terraform Enterprise Upgrades article.

• Backup

  It is imperative to backup the data of the current Terraform Enterprise installation to avoid unexpected failures which may cause system outage or, in the worst case scenario, data loss. Terraform Enterprise offers a number of options for backup and restore method depending on operation mode, however, the common method available across operation modes is Bakup/Restore API. When creating a backup via API, it is required to provide a password and this password will need to be the same when restore Terraform Enterprise from the backup therefore this information is crucial, it is also important to note that the restoration needs to be performed against the same version of Terraform Enterprise where the backup is created, so these backups may only be used in the event of a rollback. In addition to availability of the Backup/Restore API, the following options are also commonly suggested as alternative:

  • For mounted disk operational mode, a backup can be made of the disk mount location.
  • For external services operation mode, backup procedures can be performed against those external services directly. The exact procedures may differ depending on the cloud provider used, or how external services are setup (for example, when using on-premise external services).

Upgrade

The upgrade procedures of Terraform Enterprise are available on this public document

In the scenario where Terraform Enterprise has been installed with pinned version, the Dashboard on the administration console port 8800 should display all the higher versions with Disabled status due to current version is pinned. You are required to remove the pinned version with these steps prior to upgrade:

1. Connect to the Terraform Enterprise host using SSH.

2. Execute the following command to remove the pinned version. If it is more desirable to update the pinned version, the release sequence may be used in place of 0.

    $ replicatedctl params set ReleaseSequence --value 0

Troubleshooting Failures On Upgrade

Replicated UI display failure with the error message Container ptfe_base_startup failed: Timeout waiting for event startup sequencing - phase 1

During the upgrade, the status of the application on the Replicated dashboard will transition to the starting up application stage. In the scenario where the status eventually transition to fail with the error Container ptfe_base_startup failed: Timeout waiting for event startup sequencing - phase 1, the diagnosis steps below are recommended in order to proceed to the solution:

Symptom Diagnosis

1. The existing installation is operating on mounted disk mode
2. Navigate into Terraform Enterprise host machine via SSH
3. Execute command sudo docker logs ptfe_base_startup and review the output if matches with the output below

2020/11/13 13:23:38 version: 91a42b2
2020/11/13 13:23:38 setting default postgres connect_timeout to 30
2020/11/13 13:23:39 amqp://rabbitmq.ptfe:5672: not ready yet: Exception (501) Reason: "read tcp 172.17.0.18:53612->172.17.0.1:5672: read: connection reset by peer"
2020/11/13 13:23:39 redis://redis:6379: ready
2020/11/13 13:23:39 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:40 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:40 amqp://rabbitmq.ptfe:5672: ready
2020/11/13 13:23:41 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:42 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:43 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:44 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:45 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:46 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"

4. Execute command sudo docker logs db_pw_migrate and review the output if matches with the output below

2020-11-13T13:23:48.505522000Z [+] executing password migration logic...                             
2020-11-13T13:23:48.565785000Z psql: FATAL:  password authentication failed for user "hashicorp"     
2020-11-13T13:23:48.566170000Z [+] updating postgres password...                                     
2020-11-13T13:23:48.617737000Z psql: FATAL:  password authentication failed for user "hashicorp"

Solution

If these indicators are all matched, the problem comes from incorrect internal PostgreSQL password being used, the How To Recover PostgreSQL Password on Mounted Disk guide is provided in order to recover the password.

Additional information

If the issue continues to occur after completing the steps in this guide, please contact HashiCorp Support to request further assistance.