What is Vault Eventual Consistency?
Vault enterprise applies an eventual consistency model. Only one Vault node (the leader) can write to storage at any given time. Users generally expect read-after-write consistency: in other words, after writing
foo=1, a subsequent read of
foo should return
1. Depending on how Vault is configured this isn't always the case. When using performance standbys with Integrated Storage, or when using performance replication, there are some sequences of operations that don't always yield read-after-write consistency.
A customer was facing an issue when they configured the static role for the Oracle DB secret engine for one of their applications which is used to generate the password. Later, the application reads the password via API calls. When the application team was hitting the
GET credential API, they were frequently getting
ERROR:400. As per the customer, 8 requests out of 10 were failing.
It was figured out that the application team were issuing the
GET secret request immediately after it was generated and written which was against Vault's eventual consistency model.
Customer had to introduced a small
retry in their code/process that attempts to fetch the secret with the token so that Vault takes its time to synchronise.
On the other hand, if it is a short time running process, you can use Batch-token alternatively instead of Service-token. The lease attached to a Batch-token is not written to the disk at all and so its never synced across all other performance standby nodes. This way you can authenticate to Vault without worrying about eventual consistency.
For long running processes, service tokens are a better option.
Vault v1.10 has a feature for Service-tokens to return additional metadata as part of the Service-token back to the client. The additional metadata provides a reference to the authentication token. Client attempts to connect to any of the performance standby node (not yet synced) are then automatically forward to the leader node. No client configurations is required to enable this and the noticeable difference here is via a prefix on all tokens such as:
|Token Type||Old Prefix||New Prefix|
The other change will be the size of the token which will change from 26 bytes to 92 bytes.