Unable to modify DB Secret Backend connection after password has been rotated using Vault Terraform Provider

Introduction

Problem

When attempting to update an argument using the Vault provider Database Secret Backend connection resource, the database may return a connection refused error when executing a Terraform apply. 

Example Arguments which may need a modification in vault:

• max_open_connections
• allowed_roles

While none of these arguments specifically change the connection_url argument, making an update to the terraform configuration may cause the connection to be re-checked. This may fail if the database password has been rotated as recommended by the Vault documentation. 

Example Error:

vault_database_secret_backend_connection.postgres: Modifying... [id=database/config/postgres]
╷
│ Error: error configuring database connection "database/config/postgres": Error making API request.
│
│ URL: PUT http://<Vault_URL>:8200/v1/database/config/postgres
│ Code: 400. Errors:
│
│ * error creating database object: error verifying connection: failed to connect to `host=localhost user=root database=postgres`: failed SASL auth (FATAL: password authentication failed for user "root" (SQLSTATE 28P01))
│
│ with vault_database_secret_backend_connection.postgres,
│ on main.tf line 16, in resource "vault_database_secret_backend_connection" "postgres":
│ 16: resource "vault_database_secret_backend_connection" "postgres" {

Prerequisites

• Vault Terraform Provider with vault_database_secret_backend_connection resource. 

Cause

This issue may be caused by the rotation of the initial database password configured in the provider. Once the database connection configuration is set in Terraform, the password argument can be removed, as it is no longer known by anyone other than Vault. Any modifications to attributes within the resource will trigger a recheck of the credentials set within the Vault resource. 

Example Terraform apply initiating a change to max_open_connections:

$ terraform apply
vault_mount.db: Refreshing state... [id=database]
vault_database_secret_backend_connection.postgres: Refreshing state... [id=database/config/postgres]
vault_generic_endpoint.rotate_initial_db_password: Refreshing state... [id=database/rotate-root/postgres]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
~ update in-place

Terraform will perform the following actions:

# vault_database_secret_backend_connection.postgres will be updated in-place
~ resource "vault_database_secret_backend_connection" "postgres" {
id = "database/config/postgres"
name = "postgres"
# (6 unchanged attributes hidden)

~ postgresql {
~ max_open_connections = 100 -> 5
# (6 unchanged attributes hidden)
}
}

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

vault_database_secret_backend_connection.postgres: Modifying... [id=database/config/postgres]
╷
│ Error: error configuring database connection "database/config/postgres": Error making API request.
│
│ URL: PUT http://VAULT_ADDR:8200/v1/database/config/postgres
│ Code: 400. Errors:
│
│ * error creating database object: error verifying connection: failed to connect to `host=localhost user=root database=postgres`: failed SASL auth (FATAL: password authentication failed for user "root" (SQLSTATE 28P01))
│
│ with vault_database_secret_backend_connection.postgres,
│ on main.tf line 16, in resource "vault_database_secret_backend_connection" "postgres":
│ 16: resource "vault_database_secret_backend_connection" "postgres" {

Solution:

Modify the terraform configuration file to remove the password. 

Example resource configuration which causes the issue:

resource "vault_database_secret_backend_connection" "postgres" {
  backend = vault_mount.db.path
  name = "postgres"
  allowed_roles = ["*"]
  postgresql {
    connection_url = "postgresql://{{username}}:{{password}}@localhost:5432/postgres"
    username = "root"
    password = "rootpassword"
    max_open_connections = "5"
  }
}

Removing the password argument from the connection_url block allows other attributes to be modified without rechecking the database connection. 

Example working configuration:

resource "vault_database_secret_backend_connection" "postgres" {
  backend = vault_mount.db.path
  name = "postgres"
  allowed_roles = ["*"]
    postgresql {
    connection_url = "postgresql://{{username}}:{{password}}@localhost:5432/postgres"
    username = "root"
    max_open_connections = "5"
 }
}

Outcome

After this change, execute a terraform plan and terraform apply. This should not trigger a recheck of the database connection and allow resource arguments to be updated. 

Related Article

Rotate your database root credentials using the Vault Terraform provider.