Error: Failed to query available provider packages

Problem Description

Performing a terraform run and seeing an error like this:

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/helm: 
could not connect to registry.terraform.io: Failed to request discovery 
document: Get "https://registry.terraform.io/.well-known/terraform.json": EOF

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider 
hashicorp/google: could not connect to registry.terraform.io: Failed to 
request discovery document: Get 
"https://registry.terraform.io/.well-known/terraform.json": EOF

Prerequisites

• You're on a Terraform 0.13.0 - 1.0.0 release.
  • NOTE: If you're on Terraform 1.0.1+ and running into this, please refresh your DNS settings or point to a popular public DNS server like 8.8.8.8
• This issue primarily affects those running Terraform on a MacOS/Linux/Unix platform.

Possible Causes

There are primarily 3 possible reasons for this error:

1). Skipping major terraform versions without following the upgrade guides [a].

2). Not specifying provider sources in your required_providers block [b] AND modules - if you're on a 0.13+ release (also watching out for typos in your provider declarations).

3). Faulty Golang DNS resolution on UNIX platforms. This one I'll explain more in detail below - in case that's what you're running into (& provided the above two points are satisfied).

This particular issue boils down to Go's cross-platform DNS client [c] that messes up Terraform's DNS lookups (Terraform is built using Golang) when looking up provider plugins for you, especially when using a VPN.

Provider creators (e.g Google, Helm & Kubernetes) use different DNS resolution techniques for their provider source domains - which Go's DNS client does not take into account for MacOS/Linux/Unix platforms, and instead just sticks to /etc/resolv.conf which isn't always enough.

• NOTE: You don't see the issue when you run curl against our registry because curl uses your OS's native resolver chain. If you suspect something is wrong with our Registry website, please check our Status website for confirmation.

This upstream issue affects Terraform 0.13.0 - 1.0.0 releases; and was finally fixed in Terraform 1.0.1 [d][e] when we decided to perform native Terraform builds for each OS platform as opposed to depending on Golang's cross-platform DNS client.

• It took that long because we (& other members of the OSS community) tried getting Golang's maintainers to fix the issue [c] in their DNS client, but I'm afraid it's still open at the time of this article's writing.

Possible Solutions

With that said, the 3 most-popular workarounds are to:

• build/compile terraform from source [f],
• let your OS's native DNS server do the resolution for you, i.e removing any custom DNS servers you might have [g], OR
• subscribe to a popular public DNS server e.g 8.8.8.8 [h].

[Optional] You can also see references [i-k] for more threads on how other customers have run into this issue and resolved them.

Hope that helps! For further questions on this, please feel free to chat with the Terraform community on our Discuss forums or open a Support ticket. Thank you.

Link References

[a] https://www.terraform.io/language/upgrade-guides
[b] https://www.terraform.io/language/providers/requirements#requiring-providers
[c] https://github.com/golang/go/issues/12524
[d] https://github.com/hashicorp/terraform/issues/26532#issuecomment-867790814
[e] https://github.com/hashicorp/terraform/blob/v1.0/CHANGELOG.md#101-june-24-2021
[f] https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform
[g] https://github.com/hashicorp/terraform/issues/26532#issuecomment-870835655
[h] https://github.com/hashicorp/terraform/issues/26532#issuecomment-887842128
[i] https://github.com/hashicorp/terraform/issues/26716
[j] https://github.com/hashicorp/terraform/pull/5925
[k] https://github.com/hashicorp/terraform/issues/26086