"cert-manager" adds certificates and certificate issuers as resource types in Kubernetes clusters and simplifies the process of obtaining, renewing, and using those certificates.
It can issue certificates from a variety of supported sources, including Let's Encrypt, HashiCorp Vault, and Venafi as well as private PKI.
It will ensure certificates are valid and up to date, and attempt to renew certificates at a configured time before expiry.
------------------------------------------------------------------------------------------------------
The purpose of this article is to detail the existence of a defect within the cert-manager code as of the version v1.10.0 that was introduced due to a change in the Vault API included as a dependency, as well as a workaround. Note that per https://github.com/cert-manager/cert-manager/releases/tag/v1.11.0 the 1.11.0 release of cert-manager includes a fix for this issue so we recommend an upgrade as the preferred option:
- Fixes a bug that caused the Vault issuer to omit the Vault namespace in requests to the Vault API. (#5591, @wallrj)
Prior to this issue being resolved cert-manager would omit the X-Vault-Namespace header value on requests sent to Vault, meaning any namespaced requests would instead be directed to the root Vault namespace and likely result in authentication failure/certificate issuance failure.
The behavior of the vault clients changed in the Vault API versionv1.11.0with this commit. The impact of this change means that any headers that are set in the request object passed in toRawRequestare wiped out if there are no headers set in the vault client object.
Before vault APIv1.11.0RawRequestrespected headers that were passed in via the request argument. For example:
spec:
vault:
namespace: your_namespace_name
path: path_to_access_any_mount
server: https://127.0.0.1:8200/
caBundle: #base64 encoded CA bundle
auth:
appRole:
path: approle
roleId: #plaintext role ID
secretRef:
name: approle_secret_name
key: secretId
As you can see that within the Issuer Spec of Cert-Manager, it's trying to log in to Vault using the approle auth method that's configured inside a namespace called 'your_namespace_name', however, this particular spec will fail to log in to Vault and instead report the following:
URL: POST https://127.0.0.1:8200/v1/auth/approle/login
Code: 400. Errors:
* invalid role ID
Observed Generation: 1
Reason: VaultError
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ErrInitIssuer 20s (x3 over 25s) cert-manager-issuers Error initializing issuer: error logging in to Vault server: Error making API request.
URL: POST https://127.0.0.1:8200/v1/auth/approle/login
Code: 400. Errors:
* invalid role ID
The reason for this, as explained above is the namespace parameter that is defined within spec: vault is skipped while parsing when using Cert Manager Version: v1.10.0 and above (prior to the release of v1.11.0).
A workaround to be considered is to amend the namespace name into the path: such as:
spec:
vault:
path: your_namespace_name/path_to_access_any_mount
server: https://127.0.0.1:8200/
caBundle: #base64 encoded CA bundle
auth:
appRole:
path: approle
roleId: #plaintext role ID
secretRef:
name: approle_secret_name
key: secretId
Vault namespaces are flat path-based objects so usage of either a header value/environment variable is possible, or a direct amendment of the URI/path is.
Once updated the Issuer should move into a Ready state of True.