Change in Vault 1.13.0 causes Terraform Vault Provider data source vault_aws_access_credentials to null out STS credentials

Problem:

After upgrading Vault to v1.13.0 or v.1.13.1, previously working Terraform runs which make use of the Vault provider vault_aws_access_credential data source, fail with authentication errors. 

This occurs when using any version of the Terraform Vault Provider with Vault OSS/Enterprise Version 1.13.0 or 1.13.1.

Example of the vault_aws_access_credentials data source in use with type sts. 

provider "vault" {
}
data "vault_aws_access_credentials" "creds" {
  backend = "aws"
  role = "aws-devops"
  type = "sts"
}

Symptoms:

There are no symptoms from the perspective of the vault provider as no errors are produced in the run log.

[INFO] provider.terraform-provider-vault_v3.8.0_x5: 2023/05/01 17:00:00 [DEBUG] Read "aws/sts/aws-devops" from Vault: timestamp=2023-05-01T17:00:00.000Z
[INFO] provider.terraform-provider-vault_v3.8.0_x5: 2023/05/01 17:00:00 [DEBUG] Checking if AWS sts token "" is valid: timestamp=2023-05-01T17:00:00.000Z

The main symptom of the issue is when the AWS provider attempts to make use of the credentials that were cached from the data source, which will result in a failure to use the credentials. 

Example error when the AWS provider attempts to use the credential:

AWS Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded
"tf_proto_version=5.3
[ERROR] vertex "provider[\"registry.terraform.io/hashicorp/aws\"]" error: configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.

Cause:

Due to a change in Vault v1.13.0, leases were no longer generated when requesting Dynamic AWS STS credentials from Vault. This causes AWS STS credentials to no longer be populated in the state file. 

To confirm, review the terraform.tfstate file for null credential values populated for the vault_aws_access_credentials data source. 

Example when the issue occurs:

"resources": [
    {
      "mode": "data",
      "type": "vault_aws_access_credentials",
      "name": "creds",
      "provider": "provider[\"registry.terraform.io/hashicorp/vault\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "access_key": null,
            "backend": null,
            "id": null,
            "lease_duration": null,
            "lease_id": null,
            "lease_renewable": null,
            "lease_start_time": null,
            "namespace": null,
            "region": null,
            "role": null,
            "role_arn": null,
            "secret_key": null,
            "security_token": null,
            "ttl": null,
            "type": null
          },
          "sensitive_attributes": []
        }
      ]
    }

Solution:

The solution is to upgrade Vault to version 1.13.2 or above where the change to stop creating leases has been rolled back. No changes have been made to the Terraform Vault Provider code to resolve this issue. 

For information on how to upgrade Vault, please refer to the Vault Upgrade Standard Operating Procedure.

For Vault on Kubernetes, please see the Upgrading Vault on Kubernetes guide.