Upgrading Terraform Enterprise(Replicated)

Avatar
Kampanart Mekaumput
  • Updated

Introduction

Terraform Enterprise typically has new major version releases on a monthly schedule with new feature sets and bug fixes included. Between major releases, there are occasionally patch releases made available through the same release channel. Release notes are publicly available in the terraform-enterprise-release-notes repository, and can alternatively be found in the installer dashboard at port 8800 of the installation. HashiCorp Support encourages customers to review release notes and make sure all points are understood prior to the upgrade, or to make contact with HashiCorp Support for any inquiries.

Pre-requisites

Product Information

  • Export Terraform Enterprise Configuration

    Prior to the upgrade process, HashiCorp Support strongly recommends exporting the application configuration for reference and in case they are needed for an emergency situation. The steps below are required in order to export application configuration:

    1. Navigate into Terraform Enterprise host machine via SSH
    2. Execute command replicatedctl app-config export --hidden > /path/to/backup/tfe-settings.json
    3. Ensure there is a backup of /etc/replicated.conf and the required TLS certificates and license file.

  • Current and target version of Terraform Enterprise

    Upgrading from much older release of Terraform Enterprise than the target release may require a stepped upgrade. Please refer to our official releases page to identify the required* versions that must be installed before upgrading further.

    • Installation type

      • Online

        The online installation type will require outbound communication to external networks as described in the networking requirements document. We strongly encourage customers to review the access prior to upgrade as a critical task, as having issues with network communication during upgrade may result in longer outage period and potentially cause impact to the downstream services which rely on Terraform Enterprise.

      • Airgap

        Airgapped installations require the installation package to be downloaded as noted in our separate Considerations for Administrators Performing Airgapped Terraform Enterprise Upgrades article.

    • Backup

      It is imperative to backup the data of the current Terraform Enterprise installation to avoid unexpected failures which may cause system outage or, in the worst case scenario, data loss. Terraform Enterprise offers a number of options for backup and restore method depending on operation mode, however, the common method available across operation modes is Bakup/Restore API. When creating a backup via API, it is required to provide a password and this password will need to be the same when restore Terraform Enterprise from the backup therefore this information is crucial, it is also important to note that the restoration needs to be performed against the same version of Terraform Enterprise where the backup is created, so these backups may only be used in the event of a rollback. In addition to availability of the Backup/Restore API, the following options are also commonly suggested as alternative:

      • For mounted disk operational mode, a backup can be made of the disk mount location.
      • For external services operation mode, backup procedures can be performed against those external services directly. The exact procedures may differ depending on the cloud provider used, or how external services are setup (for example, when using on-premise external services).

    Upgrade

    The upgrade procedures of Terraform Enterprise are available on this public document

    In the scenario where Terraform Enterprise has been installed with pinned version, the Dashboard on the administration console port 8800 should display all the higher versions with Disabled status due to current version is pinned. You are required to remove the pinned version with these steps prior to upgrade:

    1. Connect to the Terraform Enterprise host using SSH.

    2. Execute the following command to remove the pinned version. If it is more desirable to update the pinned version, the release sequence may be used in place of 0.

       $ replicatedctl params set ReleaseSequence --value 0

    Troubleshooting Failures On Upgrade

    Replicated UI display failure with the error message Container ptfe_base_startup failed: Timeout waiting for event startup sequencing - phase 1

    During the upgrade, the status of the application on the Replicated dashboard will transition to the starting up application stage. In the scenario where the status eventually transition to fail with the error Container ptfe_base_startup failed: Timeout waiting for event startup sequencing - phase 1, the diagnosis steps below are recommended in order to proceed to the solution:

    Symptom Diagnosis

    1. The existing installation is operating on mounted disk mode
    2. Navigate into Terraform Enterprise host machine via SSH
    3. Execute command sudo docker logs ptfe_base_startup and review the output if matches with the output below
    2020/11/13 13:23:38 version: 91a42b2
2020/11/13 13:23:38 setting default postgres connect_timeout to 30
2020/11/13 13:23:39 amqp://rabbitmq.ptfe:5672: not ready yet: Exception (501) Reason: "read tcp 172.17.0.18:53612->172.17.0.1:5672: read: connection reset by peer"
2020/11/13 13:23:39 redis://redis:6379: ready
2020/11/13 13:23:39 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:40 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:40 amqp://rabbitmq.ptfe:5672: ready
2020/11/13 13:23:41 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:42 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:43 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:44 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:45 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:46 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"

    For Terraform Enterprise v202205-1 or later:

    Replicated UI display failure with the error message Container tfe_base_startup failed: Timeout waiting for event startup sequencing - phase 1

    During the upgrade, the status of the application on the Replicated dashboard will transition to the starting up application stage. In the scenario where the status eventually transition to fail with the error Container tfe_base_startup failed: Timeout waiting for event startup sequencing - phase 1, the diagnosis steps below are recommended in order to proceed to the solution:

    Symptom Diagnosis

    1. The existing installation is operating on mounted disk mode
    2. Navigate into Terraform Enterprise host machine via SSH
    3. Execute command sudo docker logs tfe_base_startup and review the output if matches with the output below
    2020/11/13 13:23:38 version: 91a42b2
2020/11/13 13:23:38 setting default postgres connect_timeout to 30
2020/11/13 13:23:39 amqp://rabbitmq.tfe:5672: not ready yet: Exception (501) Reason: "read tcp 172.17.0.18:53612->172.17.0.1:5672: read: connection reset by peer"
2020/11/13 13:23:39 redis://redis:6379: ready
2020/11/13 13:23:39 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:40 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:40 amqp://rabbitmq.tfe:5672: ready
2020/11/13 13:23:41 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:42 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:43 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:44 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:45 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
2020/11/13 13:23:46 postgres://postgres: not ready yet: pq: password authentication failed for user "hashicorp"
    1. Execute command sudo docker logs db_pw_migrate and review the output if matches with the output below
    2020-11-13T13:23:48.505522000Z [+] executing password migration logic...                             
2020-11-13T13:23:48.565785000Z psql: FATAL:  password authentication failed for user "hashicorp"     
2020-11-13T13:23:48.566170000Z [+] updating postgres password...                                     
2020-11-13T13:23:48.617737000Z psql: FATAL:  password authentication failed for user "hashicorp"

    Solution

    If these indicators are all matched, the problem comes from incorrect internal PostgreSQL password being used, the How To Recover PostgreSQL Password on Mounted Disk guide is provided in order to recover the password.

    Additional information

    If the issue continues to occur after completing the steps in this guide, please contact HashiCorp Support to request further assistance.

Articles in this section

See more

Related articles