DB Secrets Engine takes long time to rotate credentials

Avatar
Mike Oprea
  • Updated

Introduction

In some cases, you may notice that the DB Secret Engine takes a long time to reset credentials:

vault-1 vault 2025-04-22T11:57:55.128Z [TRACE] secrets.database.database_8ec356e4.vault-plugin-database-oracle: update user: transport="" status=started
vault-1 vault 2025-04-22T12:13:50.312Z [TRACE] secrets.database.database_8ec356e4.vault-plugin-database-oracle: update user: transport="" status=finished err=<nil> took=15m55.184293325s

If using VSO, this will affect the propagation of credentials, since you may see something like this:

kd -n dev vaultdynamicsecrets vso-dev | grep -A22 Events
Events:
  Type     Reason                   Age                   From                Message
  ----     ------                   ----                  ----                -------
  Warning  SecretSyncError          112m                  VaultDynamicSecret  Failed to sync the secret, horizon=6.381346118s, err=not rotated, handling ttl rollover bug
  Warning  SecretSyncError          112m                  VaultDynamicSecret  Failed to sync the secret, horizon=11.120811223s, err=not rotated, handling ttl rollover bug
  Warning  SecretSyncError          111m                  VaultDynamicSecret  Failed to sync the secret, horizon=16.025183388s, err=not rotated, handling ttl rollover bug
  Warning  SecretSyncError          111m                  VaultDynamicSecret  Failed to sync the secret, horizon=13.28942598s, err=not rotated, handling ttl rollover bug
  Warning  SecretSyncError          111m                  VaultDynamicSecret  Failed to sync the secret, horizon=30.438607679s, err=not rotated, handling ttl rollover bug
  Warning  SecretSyncError          110m                  VaultDynamicSecret  Failed to sync the secret, horizon=20.080091689s, err=not rotated, handling ttl rollover bug
  Warning  SecretSyncError          110m                  VaultDynamicSecret  Failed to sync the secret, horizon=1m19.560705814s, err=not rotated, handling ttl rollover bug
  Warning  SecretSyncError          108m                  VaultDynamicSecret  Failed to sync the secret, horizon=48.956641257s, err=not rotated, handling ttl rollover bug
  Warning  SecretSyncError          107m                  VaultDynamicSecret  Failed to sync the secret, horizon=30.694929023s, err=not rotated, handling ttl rollover bug
  Warning  SecretSyncError          97m (x26 over 7h46m)  VaultDynamicSecret  (combined from similar events): Failed to sync the secret, horizon=55.637273606s, err=not rotated, handling ttl rollover bug
  Normal   RolloutRestartTriggered  96m (x8 over 15h)     VaultDynamicSecret  Rollout restart triggered for {StatefulSet liferay}
  Normal   SecretRotated            96m                   VaultDynamicSecret  Secret synced, lease_id="", horizon=2h43m41s, sync_reason="ForceSync"

While VSO would show this "ttl rollover bug" error, the real indication is in the application logs, where the plugin would say how much time it took to execute the rotation.

Requirements

DB Secret Engine using any of the following plugins:

Solution

One solution for this would be to set the max_connection_lifetime value to a small time frame such as 5 minutes. This parameter is present in all the above mentioned DB Secret Engines.

The reason why this is happening is because Vault is expecting a sticky connection between the rotation period/schedule times and that connection gets closed in the meantime, forcing Vault to time out and re-establish a new connection in order to perform the rotation.

By setting this to a small value like "5m", it means that Vault will now expect that the connection is closed past that 5 minutes time frame and won't have the mentioned issue.

Articles in this section

See more