Introduction
Problem
Customers often ask if KMIP keys are being replicated by either Vault Disaster Recovery replication Vault or Performance Replication.
Prerequisites (if applicable)
- Vault Enterprise
- Vault KMIP Secrets Engine
- Any (3rd party) KMIP client
Overview of possible solutions
Solutions:
1. Performance Replication
From the PR primary:
vault read -format=json sys/replication/performance/status | jq -r '.data.mode'primaryFor both options it is required to obtain the UUID for the relevant KMIP secrets engine first, which can be obtained as follows:
Please note that for in the examples below the KMIP Secrets Engine is called kmip and resides in the Vault root namespace, depending on environment specifics, the commands used might have to modified.
vault secrets list -format=json | jq -r '."kmip/" | .uuid'54b1b78a-e210-7c4e-f295-cceb217251f1
Please note that the sys/raw endpoint is not enabled by default and has to be explicitly enabled in the Vault configuration File. The sys/raw endpoint should only be enabled for troubleshooting purposes.
Identifying the id of the KMIP scopes:
vault read sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/scope/info
Key Value
--- -----
value {"scopes":{"8T8qi":{"id":"8T8qi","name":"finance","role_id_by_name":{"accounting":"K6KbO"}},"ehRE6":{"id":"ehRE6","name":"support","role_id_by_name":{"vaultsupport":"ne9z0"}}}}Please note the following:
"id":"8T8qi","name":"finance""id":"ehRE6","name":"support"
Listing the keys for each scope:
vault list sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/8T8qi/Keys
----
XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo
vault list sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/ehRE6/Keys
----
VTb4K69tw3bjKLIhsRwPEQl20ghxwZm3Please note the keys: XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo and VTb4K69tw3bjKLIhsRwPEQl20ghxwZm3
From the PR Secondary:
vault read -format=json sys/replication/performance/status | jq -r '.data.mode'secondary
vault list sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/8T8qi/Keys
----
XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo
vault list sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/ehRE6/Keys
----
VTb4K69tw3bjKLIhsRwPEQl20ghxwZm3
The above output shows, that KMIP keys are being replicated between PR primaries and PR secondaries, unless the local flag is set while enabling the KMIP Secrets Engine. This can be confirmed as follows:
vault secrets list -format=json | jq '.["kmip/"] | .local'false
2. Disaster Recovery Replication
From the DR primary:
vault read -format=json sys/replication/dr/status | jq -r '.data.mode'primaryFor both options it is required to obtain the UUID for the relevant KMIP secrets engine first, which can be obtained as followed:
Please note that for in the examples below the KMIP Secrets Engine is called kmip and resides in the Vault root namespace, depending on environment specifics, the commands used might have to modified.
vault secrets list -format=json | jq -r '."kmip/" | .uuid'54b1b78a-e210-7c4e-f295-cceb217251f1Please note that the sys/raw endpoint is not enabled by default and has to be explicitly enabled in the Vault configuration File. The sys/raw endpoint should only be enabled for troubleshooting purposes.
Identifying the id of the KMIP scopes:
vault read sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/scope/info
Key Value
--- -----
value {"scopes":{"8T8qi":{"id":"8T8qi","name":"finance","role_id_by_name":{"accounting":"K6KbO"}},"ehRE6":{"id":"ehRE6","name":"support","role_id_by_name":{"vaultsupport":"ne9z0"}}}}Please note the following:
"id":"8T8qi","name":"finance""id":"ehRE6","name":"support"
Listing the keys for each scope:
vault list sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/8T8qi/Keys
----
XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo
vault list sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/ehRE6/Keys
----
VTb4K69tw3bjKLIhsRwPEQl20ghxwZm3
Please note the keys: XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo and VTb4K69tw3bjKLIhsRwPEQl20ghxwZm3
It is not possible to access the sys/raw path on the DR Secondary:
vault read sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/scope/infoError reading sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/scope/info: Error making API request.
URL: GET https://127.0.0.1:8200/v1/sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/scope/info
Code: 400. Errors:
* path disabled in replication DR secondary modeTherefore it is not possible to confirm the existence of the keys on the DR Secondary cluster, however after a DR Failover (Promoting A DR Secondary to DR Primary):
vault list sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/8T8qi/Keys
----
XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo
vault list sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/ehRE6/Keys
----
VTb4K69tw3bjKLIhsRwPEQl20ghxwZm3
The above output shows, that KMIP keys are being replicated between DR primaries and DR secondaries.
Outcome
The above outputs show that KMIP keys are being replicated by both Performance and Disaster Recovery Replication.
Additional Information
- Vault API Documentation sys/raw
- Vault KB Article KMIP key not found related issues
Vault Documentation KMIP secrets engine
Vault Tutorial Manage client encryption keys with Vault as a KMIP server
- Vault API Documentation KMIP secrets engine (API)
- Vault Tutorial Enable disaster recovery replication
-
Vault Tutorial Enable performance replication