This article explains the best practices for using Convergent Tokenization because it works based on the uniqueness of the plaintext that is being encoded and the expiration time given to the token generated out of the plaintext encoding operation.
Convergent Tokenization & How to use it?
By default, tokenization produces a unique token for every encode operation. This makes the resulting token fully independent of its plaintext and expiration.
However, it may be beneficial if the tokenization of a plaintext/expiration pair tokenizes consistently to the same value. In this case, one can create a convergent tokenization transformation.
When enabled at transformation creation time, Vault alters the calculation so that encoding a plaintext and expiration tokenizes to the same value every time, and storage keeps only a single entry of that token.
Let's see the following example to understand the best practices around convergent tokenization transformation:
~ % vault secrets enable transform
Success! Enabled the transform secrets engine at: transform/
~ % vault write transform/role/mobile-pay transformations="credit-card, credit-card-convergent"
Success! Data written to: transform/role/mobile-pay
~ % vault write transform/transformations/tokenization/credit-card-convergent \
allowed_roles="*" convergent=true
Success! Data written to: transform/transformations/tokenization/credit-card-convergent
~ % vault read transform/transformations/tokenization/credit-card-convergent
Key Value
--- -----
allowed_roles [*]
deletion_allowed false
mapping_mode default
max_ttl 0s ---> [No max_ttl defined]
stores [builtin/internal]
templates <nil>
type tokenization
~ % vault write transform/encode/mobile-pay value=5555-6666-7777-8888 \
transformation=credit-card-convergent ---> [This is without expiration time - so it is going to exist indefinitely --- not a best practise]
Key Value
--- -----
encoded_value DaCJhefr1oRrNW2Kxb29JTXVEco4KGtqW1kpPA5EqQcrwMw1XrXHshjbkqqQnWnj6jzWyKwcpBSx
~ % vault write transform/encode/mobile-pay value=5555-6666-7777-8888 \
transformation=credit-card-convergent ---> [This is without expiration time - so it is going to exist indefinitely --- not a best practise but returns the same encoded value as previous due to "convergent=true"]
Key Value
--- -----
encoded_value DaCJhefr1oRrNW2Kxb29JTXVEco4KGtqW1kpPA5EqQcrwMw1XrXHshjbkqqQnWnj6jzWyKwcpBSx
~ % vault write transform/encode/mobile-pay value=5555-6666-7777-8888 \
transformation=credit-card-convergent expiration="2025-01-02T00:00:00Z"
Key Value
--- -----
encoded_value AHLdmFvTRkhXKv7JHJbR1exuWU6AXuJRZWSdudcvSTfVcuZfy6EQJgsYu6BMH36QmGbPnVCMTJKejRFFaxVxjo8
~ % vault write transform/encode/mobile-pay value=5555-6666-7777-8888 \
transformation=credit-card-convergent expiration="2025-01-02T00:00:00Z"
Key Value
--- -----
encoded_value AHLdmFvTRkhXKv7JHJbR1exuWU6AXuJRZWSdudcvSTfVcuZfy6EQJgsYu6BMH36QmGbPnVCMTJKejRFFaxVxjo8 ---> [Returning the same encoded value with expiration time which is a good and recommended practice]
### Token Lookup using Expiration Time
~ % vault write transform/tokens/mobile-pay value=5555-6666-7777-8888 \
transformation=credit-card-convergent expiration="any" -format=json
{
"request_id": "af793dda-0a14-9db7-5f92-0b785559d5bd",
"lease_id": "",
"lease_duration": 0,
"renewable": false,
"data": {
"tokens": [
"DaCJhefr1oRrNW2Kxb29JTXVEco4KGtqW1kpPA5EqQcrwMw1XrXHshjbkqqQnWnj6jzWyKwcpBSx",
"AHLdmFvTRkhXKv7JHJbR1exuWU6AXuJRZWSdudcvSTfVcuZfy6EQJgsYu6BMH36QmGbPnVCMTJKejRFFaxVxjo8"
]
},
"warnings": [
"Endpoint ignored these unrecognized parameters: [-format]"
],
"mount_type": "transform"
}
~ % vault write /transform/metadata/mobile-pay value=DaCJhefr1oRrNW2Kxb29JTXVEco4KGtqW1kpPA5EqQcrwMw1XrXHshjbkqqQnWnj6jzWyKwcpBSx transformation=credit-card-convergent
Key Value
--- -----
metadata <nil> ----- [because this was generated without an expiration time]
~ % vault write /transform/metadata/mobile-pay value=AHLdmFvTRkhXKv7JHJbR1exuWU6AXuJRZWSdudcvSTfVcuZfy6EQJgsYu6BMH36QmGbPnVCMTJKejRFFaxVxjo8 transformation=credit-card-convergent
Key Value
--- -----
metadata <nil>
expiration_time 2025-01-02 00:00:00 +0000 UTC ----- [because this was generated with an expiration time]
Even if I define 'max_ttl' which is a good practise, then:
~ % vault write transform/transformations/tokenization/credit-card-convergent \
allowed_roles="*" convergent=true max_ttl=365d
Success! Data written to: transform/transformations/tokenization/credit-card-convergent
~ % vault read transform/transformations/tokenization/credit-card-convergent
Key Value
--- -----
allowed_roles [*]
deletion_allowed false
mapping_mode default
max_ttl 8760h
stores [builtin/internal]
templates <nil>
type tokenization
~ % vault write transform/encode/mobile-pay value=8888-6666-7777-8888 \
transformation=credit-card-convergent expiration="2025-01-02T00:00:00Z"
Key Value
--- -----
encoded_value AHLdmFvTRkh3EXzikEisYMPc4u91bvHxi5stoHRWvrDwASK6mjY7GQCGsU57xxQ6WUB9hnAKJPG5HA6V8Q39Jhm
akashgupta@akashgupta-FVFGJ273Q05P ~ % vault write transform/encode/mobile-pay value=8888-6666-7777-8888 \
transformation=credit-card-convergent expiration="2025-01-02T00:00:00Z"
Key Value
--- -----
encoded_value AHLdmFvTRkh3EXzikEisYMPc4u91bvHxi5stoHRWvrDwASK6mjY7GQCGsU57xxQ6WUB9hnAKJPG5HA6V8Q39Jhm
akashgupta@akashgupta-FVFGJ273Q05P ~ % vault write transform/encode/mobile-pay value=8888-6666-7777-8888 \
transformation=credit-card-convergent expiration="2025-01-02T00:00:00Z"
Key Value
--- -----
encoded_value AHLdmFvTRkh3EXzikEisYMPc4u91bvHxi5stoHRWvrDwASK6mjY7GQCGsU57xxQ6WUB9hnAKJPG5HA6V8Q39Jhm
~ % vault write transform/tokens/mobile-pay value=8888-6666-7777-8888 \
transformation=credit-card-convergent expiration="any" -format=json
{
"request_id": "7bf86dc7-4a47-ca69-24b0-d1ede8ded744",
"lease_id": "",
"lease_duration": 0,
"renewable": false,
"data": {
"tokens": [
"AHLdmFvTRkh3EXzikEisYMPc4u91bvHxi5stoHRWvrDwASK6mjY7GQCGsU57xxQ6WUB9hnAKJPG5HA6V8Q39Jhm" ----> [same value each time because the expiration time was within the max_ttl limit]
]
},
"warnings": [
"Endpoint ignored these unrecognized parameters: [-format]"
],
"mount_type": "transform"
}
If the expiration time exceedes the max_ttl then:
~ % vault write transform/encode/mobile-pay value=9999-6666-7777-8888 \
transformation=credit-card-convergent expiration="2027-01-02T00:00:00Z"
WARNING! The following warnings were returned from Vault:
* expiration "17532h23m21.081313713s" is longer than the permitted max_ttl
"8760h0m0s", so max_ttl is being used
Key Value
--- -----
encoded_value AHLdmFvTRkjLaosvAnnSpcXkCrUGNYu5JQ2u9P4M9YeVjGTVQ8FubnaEd1wZ9YUfPKC28Ux17PXEhR1Rv36SWAY
~ % vault write transform/encode/mobile-pay value=9999-6666-7777-8888 \
transformation=credit-card-convergent expiration="2027-01-02T00:00:00Z"
WARNING! The following warnings were returned from Vault:
* expiration "17532h23m17.062201807s" is longer than the permitted max_ttl
"8760h0m0s", so max_ttl is being used
Key Value
--- -----
encoded_value AHLdmFvTRkndfx2emsd8g5sqNiR4qHU2dkrZf3F4KsgTmRkmwB9W17GgP4dW62zAwWRiHrZHFYKbvxWPgbZCgqe ---> [the value changes because the overall lifetime of the token is different with a few seconds]
# Vault underlying place to lookup all the tokens generated by tokenization transformation where 'd6762492-8bf4-4877-3806-c86fc3e6fdae' is the mount UUID ("vault secrets list -detailed").
# only accessible when 'raw_storage_endpoint' is enable from the Vault config file - not recommended in production environment.
~ % vault list sys/raw/logical/d6762492-8bf4-4877-3806-c86fc3e6fdae/transformations/tokenization/credit-card-convergent_data/state/tokens/
Note: Convergent tokenization has a small performance penalty in external stores and a larger one in the built-in store due to the need to avoid duplicate entries and to update metadata when convergently encoding. It is recommended that if one has some use cases that require convergence and some that do not, one should create two different tokenization transforms with convergence enabled on only one.