Introduction
Problem
For this example MongoDB has been used and MongoDB fails to start with the following error message:
{"t":{"$date":"2025-10-30T09:40:07.725+00:00"},"s":"E", "c":"STORAGE", "id":24248, "ctx":"initandlisten","msg":"Unable to retrieve key","attr":{"keyId":".system","error":{"code":2,"codeName":"BadValue","errmsg":"KMIP get key 'XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo' failed, code: 1 error: result reason: ResultReasonItemNotFound; additional message: no object found with Unique ID \"XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo\""}}}Please note that MongoDB has been used as an example and that the issue could occur with any other (3rd party) KMIP client.
Prerequisites
- Vault Enterprise
- Vault KMIP Secrets Engine
- Any (3rd party) KMIP client
Cause
This is an example of a request to issued by a KMIP application to find / request a KMIP key, for this example mongodb has been used, the data displayed below is taken from the Vault Audit Log:
"request": {
"client_certificate_serial_number": "320846363628031116837096881145607069997040299552",
"data": {
"kmip_request": {
"tag": "RequestMessage",
"value": [
{
"tag": "RequestHeader",
"value": [
{
"tag": "ProtocolVersion",
"value": [
{
"tag": "ProtocolVersionMajor",
"type": "Integer",
"value": "0x00000001"
},
{
"tag": "ProtocolVersionMinor",
"type": "Integer",
"value": "0x00000002"
}
]
},
{
"tag": "BatchCount",
"type": "Integer",
"value": "0x00000001"
}
]
},
{
"tag": "BatchItem",
"value": [
{
"tag": "Operation",
"type": "Enumeration",
"value": "0x0000000a"
},
{
"tag": "RequestPayload",
"value": [
{
"tag": "UniqueIdentifier",
"type": "TextString",
"value": "XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo"
}
]
}
]
}
]
}
}Succesful response:
"response": {
"data": {
"kmip_response": {
"tag": "ResponseMessage",
"value": [
{
"tag": "ResponseHeader",
"value": [
{
"tag": "ProtocolVersion",
"value": [
{
"tag": "ProtocolVersionMajor",
"type": "Integer",
"value": "0x00000001"
},
{
"tag": "ProtocolVersionMinor",
"type": "Integer",
"value": "0x00000002"
}
]
},
{
"tag": "TimeStamp",
"type": "DateTime",
"value": "2025-10-30T09:35:51+02:00"
},
{
"tag": "BatchCount",
"type": "Integer",
"value": "0x00000001"
}
]
},
{
"tag": "BatchItem",
"value": [
{
"tag": "Operation",
"type": "Enumeration",
"value": "0x0000000a"
},
{
"tag": "ResultStatus",
"type": "Enumeration",
"value": "0x00000000"
},
{
"tag": "ResponsePayload",
"value": [
{
"tag": "ObjectType",
"type": "Enumeration",
"value": "0x00000002"
},
{
"tag": "UniqueIdentifier",
"type": "TextString",
"value": "XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo"
},
{
"tag": "SymmetricKey",
"value": [
{
"tag": "KeyBlock",
"value": [
{
"tag": "KeyFormatType",
"type": "Enumeration",
"value": "hmac-sha256:e8f33eb88a1b7654525daf2bba6eb539eb9af34c5a579fea48bad4329d8e0a23"
},
{
"tag": "KeyValue",
"value": [
{
"tag": "KeyMaterial",
"type": "ByteString",
"value": "hmac-sha256:a41429d1945da28a4eda5e0e60cf9575de45b7946b9f0b758ea8ff364acfa386"
}
]
},
{
"tag": "CryptographicAlgorithm",
"type": "Enumeration",
"value": "0x00000003"
},
{
"tag": "CryptographicLength",
"type": "Integer",
"value": "0x00000100"
}
]
}
]
}
]
}
]
}
]
}
}
}Failed response:
"response": {
"data": {
"kmip_response": {
"tag": "ResponseMessage",
"value": [
{
"tag": "ResponseHeader",
"value": [
{
"tag": "ProtocolVersion",
"value": [
{
"tag": "ProtocolVersionMajor",
"type": "Integer",
"value": "0x00000001"
},
{
"tag": "ProtocolVersionMinor",
"type": "Integer",
"value": "0x00000002"
}
]
},
{
"tag": "TimeStamp",
"type": "DateTime",
"value": "2025-10-30T10:43:29+02:00"
},
{
"tag": "BatchCount",
"type": "Integer",
"value": "0x00000001"
}
]
},
{
"tag": "BatchItem",
"value": [
{
"tag": "Operation",
"type": "Enumeration",
"value": "0x0000000a"
},
{
"tag": "ResultStatus",
"type": "Enumeration",
"value": "0x00000001"
},
{
"tag": "ResultReason",
"type": "Enumeration",
"value": "0x00000001"
},
{
"tag": "ResultMessage",
"type": "TextString",
"value": "result reason: ResultReasonItemNotFound; additional message: no object found with Unique ID \"XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo\""
}
]
}
]
}
}
}
Overview of possible solutions
Solutions:
Using the Vault KMIP secrets engine, KMIP keys are created under a specific KMIP scope. When a client certificate used for KMIP authentication by a KMIP client is issued under a different scope, the KMIP client is not able to find the certificate while it actually does exist in Vault.
In the above example 2 scopes exist:
finance
support
The KMIP key has originally been issued using a role which belongs to the finance scope, therefore the key can only be found by a KMIP client if a certificate is used for client authentication which has been generated by a role which belongs to the finance scope.
The successful response is an example of a client certificate used issued under the finance scope
The failed response is an example of a client certificate used under the support scope.
Currently two options exist to confirm if KMIP keys exist and under which KMIP key has been created.
For both options it is required to obtain the UUID for the relevant KMIP secrets engine first, which can be obtained as follows:
Please note that for in the examples below the KMIP Secrets Engine is called kmip and resides in the Vault Root namespace, depending on environment specifics, the commands used might have to modified.
vault secrets list -format=json | jq -r '."kmip/" | .uuid'54b1b78a-e210-7c4e-f295-cceb217251f11. sys/raw endpoint
Please note that the sys/raw endpoint is not enabled by default and has to be explicitly enabled in the Vault Configuration File. The sys/raw endpoint should only be enabled for troubleshooting purposes.
Identifying the id of the KMIP scopes:
vault read sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/scope/infoKey Value
--- -----
value {"scopes":{"8T8qi":{"id":"8T8qi","name":"finance","role_id_by_name":{"accounting":"K6KbO"}},"ehRE6":{"id":"ehRE6","name":"support","role_id_by_name":{"vaultsupport":"ne9z0"}}}}Please note the following:
"id":"8T8qi","name":"finance""id":"ehRE6","name":"support"
Listing the keys per scope:
vault list sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/8T8qi/
Keys
----XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo
vault list sys/raw/logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/ehRE6Keys
----
VTb4K69tw3bjKLIhsRwPEQl20ghxwZm3
Please note that key XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo only exists under the scope 8T8qi/while a different key existing under the ehRE6/ scope.
2. Vault Snapshots
Generating a Vault Snapshot:
vault operator raft snapshot save /tmp/vault.snapUsing vault raft snapshot inspect and grep for the uuid of the KMIP secrets egine:
vault operator raft snapshot inspect -depth=5 /tmp/vault.snap | grep -i 54b1b78a-e210-7c4e-f295-cceb217251f1
1 270B logical/54b1b78a-e210-7c4e-f295-cceb217251f1/ca 1 1.7KB
logical/54b1b78a-e210-7c4e-f295-cceb217251f1/client_certs/8T8qi/K6KbO 1 763B
logical/54b1b78a-e210-7c4e-f295-cceb217251f1/client_certs/ehRE6/ne9z0 1 767B
logical/54b1b78a-e210-7c4e-f295-cceb217251f1/config 1 343B
logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/8T8qi/XMfL8IIi8yYbXXTECnujDaPJeq9KEXTo 1 721B
logical/54b1b78a-e210-7c4e-f295-cceb217251f1/managed-objects/ehRE6/VTb4K69tw3bjKLIhsRwPEQl20ghxwZm3 1 721B
logical/54b1b78a-e210-7c4e-f295-cceb217251f1/scope/8T8qi/role 1 218B
logical/54b1b78a-e210-7c4e-f295-cceb217251f1/scope/ehRE6/role 1 218B
logical/54b1b78a-e210-7c4e-f295-cceb217251f1/scope/info The only possible issue with the vault raft snapshot inspect approach, is mapping the KMIP scope id to a name, this is currently not possible. However this approach can be used to confirm that the requested key does actually exist.
Outcome
By ensuring that the KMIP client is using a client certificate which has been issued using a KMIP role, for the same KMIP scope under which the KMIP key exists and which was originally used when the KMIP key was generated by the KMIP client the KMIP client should be able to find and use the KMIP key it is looking for.
Additional Information
- Vault API Documentation sys/raw
- Vault KB Article KMIP key replication
- Vault KB Article Renew KMIP Certificates
Vault Documentation KMIP secrets engine
Vault Tutorial Manage client encryption keys with Vault as a KMIP server
-
Vault API Documentation KMIP secrets engine (API)