Introduction
In some cases, you may notice that the DB Secret Engine takes a long time to reset credentials:
vault-1 vault 2025-04-22T11:57:55.128Z [TRACE] secrets.database.database_8ec356e4.vault-plugin-database-oracle: update user: transport="" status=started
vault-1 vault 2025-04-22T12:13:50.312Z [TRACE] secrets.database.database_8ec356e4.vault-plugin-database-oracle: update user: transport="" status=finished err=<nil> took=15m55.184293325sIf using VSO, this will affect the propagation of credentials, since you may see something like this:
kd -n dev vaultdynamicsecrets vso-dev | grep -A22 Events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning SecretSyncError 112m VaultDynamicSecret Failed to sync the secret, horizon=6.381346118s, err=not rotated, handling ttl rollover bug
Warning SecretSyncError 112m VaultDynamicSecret Failed to sync the secret, horizon=11.120811223s, err=not rotated, handling ttl rollover bug
Warning SecretSyncError 111m VaultDynamicSecret Failed to sync the secret, horizon=16.025183388s, err=not rotated, handling ttl rollover bug
Warning SecretSyncError 111m VaultDynamicSecret Failed to sync the secret, horizon=13.28942598s, err=not rotated, handling ttl rollover bug
Warning SecretSyncError 111m VaultDynamicSecret Failed to sync the secret, horizon=30.438607679s, err=not rotated, handling ttl rollover bug
Warning SecretSyncError 110m VaultDynamicSecret Failed to sync the secret, horizon=20.080091689s, err=not rotated, handling ttl rollover bug
Warning SecretSyncError 110m VaultDynamicSecret Failed to sync the secret, horizon=1m19.560705814s, err=not rotated, handling ttl rollover bug
Warning SecretSyncError 108m VaultDynamicSecret Failed to sync the secret, horizon=48.956641257s, err=not rotated, handling ttl rollover bug
Warning SecretSyncError 107m VaultDynamicSecret Failed to sync the secret, horizon=30.694929023s, err=not rotated, handling ttl rollover bug
Warning SecretSyncError 97m (x26 over 7h46m) VaultDynamicSecret (combined from similar events): Failed to sync the secret, horizon=55.637273606s, err=not rotated, handling ttl rollover bug
Normal RolloutRestartTriggered 96m (x8 over 15h) VaultDynamicSecret Rollout restart triggered for {StatefulSet liferay}
Normal SecretRotated 96m VaultDynamicSecret Secret synced, lease_id="", horizon=2h43m41s, sync_reason="ForceSync"While VSO would show this "ttl rollover bug" error, the real indication is in the application logs, where the plugin would say how much time it took to execute the rotation.
Requirements
DB Secret Engine using any of the following plugins:
Solution
One solution for this would be to set the max_connection_lifetime value to a small time frame such as 5 minutes. This parameter is present in all the above mentioned DB Secret Engines.
The reason why this is happening is because Vault is expecting a sticky connection between the rotation period/schedule times and that connection gets closed in the meantime, forcing Vault to time out and re-establish a new connection in order to perform the rotation.
By setting this to a small value like "5m", it means that Vault will now expect that the connection is closed past that 5 minutes time frame and won't have the mentioned issue.