Using custom and community providers in Terraform Cloud and Enterprise

Use Case

The use of custom and community providers in Terraform Cloud and Terraform Enterprise is outlined in our Custom and Community Providers documentation. There are 4 options :

• utilize Private Module Registry
• place the provider binary in the repository
• build a custom Terraform bundle (Terraform Enterprise only).
• create and use Network Mirror 

Procedure

Private Module Registry

The recommended and preferred way. Historically the name of the service was "Private Module Registry", or PMR for short. But with evolving of the functionality it now includes support for versioning and a searchable list of available providers and modules.  Terraform Cloud's private registry works similarly to the public Terraform Registry and helps you share Terraform providers and Terraform modules across your organization.

• The detailed manual for publishing providers can be found here
• A manual with of usage can be accessed on this page (geared more towards end-user)

This feature available in Terraform Cloud natively and present in Terraform Enterprise from v202112-2

Using Git submodules for commonly used providers

When the same provider is used across many repositories, we recommend using Git submodules to avoid the need to place custom providers in each of your configuration repositories. To do so, use the following steps:

1. Create a separate repository for your custom provider (the “provider repository”).
2. Compile a linux_amd64 binary for your provider and make it executable using chmod +x.
3. Upload the binary to the root of your provider repository.
4. Both Terraform Cloud and Terraform Enterprise check for custom providers in a path relative to the location where Terraform is run. Navigate to the root of your configuration repository, or the Terraform Working Directory.
5. Add a submodule to the configuration repository that references the provider repository with the following command, replacing $ORGANIZATION and $REPOSITORY with the VCS organization and repository that the provider is in:

Note: Since there is no way to pass HTTPS credentials during a Terraform run, the following method assumes that the user is cloning a publicly available repository. If you are instead using a repository that is not publicly available, the user must instead provide SSH keys in the VCS connection settings and clone over SSH.

$ git submodule add \
    https://github.com/$ORGANIZATION/$REPOSITORY.git \
    terraform.d/plugins/linux_amd64

Workspaces with custom providers must have Include submodules on clone enabled in order for the custom or community provider to be downloaded. This option can be found in a Workspace’s settings under “Version Control”.

Note: When a provider binary is updated, the submodule in any configuration repository that references it will also need to be updated.

Using a custom Terraform bundle

When using Terraform Enterprise, it is also possible to include custom and community providers in a custom Terraform bundle. The steps to set up the terraform-bundle CLI tool are outside of the scope of this document, so the below steps will assume that terraform-bundle has already been compiled and is in the $PATH.

1. Within the providers block of the terraform-bundle.hcl configuration file, add the custom provider with the following format.

     providers {
       my-custom-provider = ["1.0"]
      }

1. Create a ./plugins subdirectory in the directory that terraform-bundle will be run in. Optionally, you can use the -plugin-dir flag to specify a location of where to find the provider.

2. Place the custom or community provider in the ./plugins directory. To be recognized, custom providers must have a name following the form of terraform-provider-<NAME>_v<VERSION>. In addition, ensure that the provider is built using the same operating system and architecture used for Terraform Enterprise. Typically this will be linux_amd64.

Using network mirror

Provider network mirrors are supported only in Terraform CLI v0.13.2 and later. Prior versions do not support this protocol.

If there is a possibility to support additional web-services and related infrastructure,  Network Mirror protocol can be used. Basically, you will need to create and HTTP-based service in a specific format. Most Terraform-native services use the remote service discovery protocol so that the physical location of the endpoints can potentially be separated from the hostname used in identifiers. The Provider Network Mirror protocol does not use the service discovery indirection, because a network mirror location is only a physical location and is never used as part of the identifier of a dependency in a Terraform configuration.

Instead, the provider installation section of the CLI configuration accepts a base URL directly. The given URL must use the scheme https:, and should end with a trailing slash so that the relative URLs of the individual operation endpoints will be resolved beneath it. 

provider_installation {
    network_mirror {
        url = "https://terraform.example.com/providers/"
    }
}

So in this case,  source code should be also modified to use the mirror.  This method is one of the least recommended, as it requires a lot of additional steps and upkeep overhead.

Detailed manual available on this page.

Additional Information

Eventually, 3 last steps may not be necessary, and can be replaced with Providers in the Terraform Registry.  But,  all infrastructures differ so there is place for every approach.