Terraform apply failed with "no suitable version installed" or "Could not load plugin" in Terraform Enterprise

Introduction

Terraform Enterprise initiates plan and apply operations in isolated environments (containers) therefore the .terraform directory initialized by the Terraform plan and apply operations are completely isolated. There are possibilities that the provider plugins installed by terraform init of the plan execution are not installed by terraform init of the apply execution. This article highlights the symptom with the steps provided on how to handle the failure no suitable version installed or Could not load plugin. Please note that the error Could not load plugin is specific to Terraform CLI 0.13 and later which can also be caused by the upgrade from 0.12, if you are upgrading from version 0.12 and encounter the error Could not load plugin, please visit the Plugin reinitialization error after upgrade to Terraform 0.13 article for further information.

Problem

The Terraform plan environment initializes the working directory with the terraform init command using remote backend, in contrast to the Terraform apply environment in which initializes the working directory with the init command without any backend. As a result, .terraform directory in the apply environment will only have the provider plugins that can be worked out from Terraform configuration files only. In a scenario where all resources that are associated with a provider are removed completely from the configuration files, that particular provider will not be installed as part of terraform init execution in the apply environment and leads into the error no suitable version installed on Terraform CLI version 0.12 and lower or Could not load plugin on the version 0.13 and later

Procedure

Terraform CLI 0.12.20 and later

1. Examine the apply output to find the required provider, which appears in front of the no suitable version installed error message or at the bottom of the output in case of Could not load plugin

2. Create a separate .tf file and add required_providers block for the required provider. Please note that the source argument is only supported in Terraform CLI 0.13 and later.

3. Queue the run and apply, if no other error and the apply succeeded, the latest state version should contain no reference to the destroyed resources.

4. Remove the .tf file created in step 2) and the workspace should now be back into the clean state.

Terraform CLI 0.12.19 and below

Due to Terraform required_providers configuration block is not available, it will require Terraform state manipulation described in the Modifying Terraform states in Terraform Enterprise or Terraform Cloud article to remove the resources from the state and clean up the backend infrastructure resources manually.