Database secrets engine configuration fails with i/o timeout

Introduction

The database secrets engine requires configuration of the proper plugin and connection information as outlined in the setup section of the documentation. During the configuration, Vault verifies the connection to the database and saves the information if the connection was successful.

This article captures a failure of the connection verification which could occur when Performance Replication is used.

Problem

The database connection verification process that is performed during configuration differs, depending on where the configuration request is made. 

In case the vault write database/config/connection1 command is executed on the primary cluster, Vault will only verify whether the primary cluster is able to connect to the database. The connection verification will be successful and connection information saved.

In case the vault write database/config/connection1 command is executed on the secondary cluster, Vault will verify whether both the primary and secondary cluster is able to connect to the database. If either one of the clusters are unable to connect to the database, an error will be returned stating a connection issue in the form of an i/o timeout.

Prerequisites

• Vault Enterprise (All versions) with Performance Replication enabled.

Cause

Due to the design of Performance Replication, the database secrets engine connection details are first written to the primary cluster and then replicated to the secondary cluster.

When a write call is made to the primary cluster there is no need to verify whether the secondary cluster is able to connect to the database, as the connection information will be replicated regardless.

In case the same write call is made from the secondary cluster, the write is forwarded to the primary and since the default is to verify the connection, both the secondary and primary tries to establish a connection to the database.

If the verification from secondary is successful, but fails from the primary an error will be returned.

Please note that due to a current defect, the error wrongly states that both the primary and secondary clusters are unable to connect. (Defect VAULT-15023)

For Example:

vault write database/config/connection1 \ 
  plugin_name=mssql-database-plugin \
  connection_url='sqlserver://{{username}}:{{password}}@172.21.X.X:1433' \
  allowed_roles="my-role" \
  username="sa" \
  password="yourStrong@Password"

Error writing data to database/config/connection1: Error making API request.

URL: PUT http://127.0.0.1:8300/v1/database/config/connection1
Code: 400. Errors:

* 2 errors occurred:
* errors from both primary and secondary; primary error was error creating database object: error verifying connection: unable to open tcp connection with host '172.21.X.X:1433': dial tcp 172.21.X.X:1433: i/o timeout; secondary errors follow
* error creating database object: error verifying connection: unable to open tcp connection with host '172.21.X.X:1433': dial tcp 172.21.X.X:1433: i/o timeout

Overview of possible solutions

Solutions:

Any of the following:

• Ensure that both the primary and secondary clusters are able to communicate/connect to the database being configured.

• Mark the secrets engine as local-only by specifying the -local parameter. This disables replication for the specified mount.

While the credential verification can be disabled in order to have the settings written, the same error may occur when adding a role. Disabling credential verification for a role is not possible.

• Disable the connection verification parameter when writing the connection information. This can be achieved by setting verify_connection=false

Outcome

Configuration of the database plugin and connection information are successful and returns no errors.

Additional Information

• Vault Documentation: Databases
• Vault Documentation: Vault Enterprise Replication
• Vault API doc: verify_connection
• Vault API doc: local