Consul-Vault Secrets Backend OpenSSL Certificate Request/Approval for Local K8s Clusters (KinD | Minikube) – HashiCorp Help Center

The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Purpose:

Guide for establishing a Vault secrets backend for Consul in a local KinD or Minikube test environment with Vault TLS enabled certificate authentication.

Note: This is not intended for production use, but for local testing and learning labs.

Prerequisites:

Operable Kubernetes KinD or Minikube cluster
kubectl context set to KinD or Minikube cluster
OpenSSL or LibreSSL Command Line Tool >= v2.8.3 Installed - Quick Start

user@macbook ~$ which openssl
/usr/bin/openssl

user@macbook ~$ openssl version
LibreSSL 2.8.3

Certificate Request Environmental Variables:

Typically these variables shouldn't need to change, but they can be altered to suit your specific use-case of Vault. Set these environment variables in your current shell/bash/zsh terminal session.

NAMESPACE: The Kubernetes Namespace Vault will be deployed to.
SERVICE: Name of the Vault service. Defaults to 'vault'
SECRET_NAME: Name of Kubernetes generic secret to create in the secrets store.
CSR_NAME: Name of certificate signing request as seen by Kubernetes.
TMPDIR: Certificate destination temporary working directory.

export NAMESPACE="vault"
export SERVICE="vault"
export SECRET_NAME="vault-server-tls"
export TMPDIR="./cert"
export CSR_NAME="vault-csr"

Generate Vault TLS Certificate Key

This CSR Configuration File establishes the Vault TLS Certificate Key Usages, and approved Subject Alternative Names to include in the Kubernetes CA approved Vault Cluster TLS Certificate.

( cat <<-EOF
[ req ]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[ req_distinguished_name ]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${SERVICE}
DNS.2 = ${SERVICE}.${NAMESPACE}
DNS.3 = ${SERVICE}.${NAMESPACE}.svc
DNS.4 = ${SERVICE}.${NAMESPACE}.svc.cluster.local
DNS.5 = ${SERVICE}-agent-injector-svc
DNS.6 = ${SERVICE}-agent-injector-svc.${NAMESPACE}
DNS.7 = ${SERVICE}-agent-injector-svc.${NAMESPACE}.svc
DNS.8 = ${SERVICE}-agent-injector-svc.${NAMESPACE}.svc.cluster.local
IP.1 = 127.0.0.1
EOF
) | tee "$TMPDIR/csr.conf"

Generate OpenSSL PEM Encoded CSR for Kubernetes API

Using the previously generated Certificate Signing Request (CSR) Configuration File, we'll use OpenSSL to generate a PEM encoded request to submit to the Kubernetes API for approval.

# Generate K8s server.csr for submittal to K8s API
openssl req \
  -new \
  -key "$TMPDIR/vault.key" \
  -subj "/O=system:nodes/CN=system:node:$SERVICE.$NAMESPACE.svc" \
  -out "$TMPDIR/server.csr" \
  -config "$TMPDIR/csr.conf"

# Set the required server.csr file permissions.
sudo chmod 400 "$TMPDIR/server.csr"

Kubernetes API Signers Discussion:

As of Kubernetes v1.22+, the -subj argument flag denoting the system:nodes and system:node: K8s Core Component Roles are required for the CSR Approval Process to Kubernetes via the Kubernetes API kubernetes.io/kubelet-serving signer. These let the Kubernetes API know that this certificate request, upon approval, will issue a certificate that will provide access to resources required by the kubelet, including read access to all secrets, and write access to all pod status objects. The Kubernetes API will not approve the certificate nor distribute it should these specific -subj arguments fail to denote the system:nodes and system:node component roles.

Generate and Apply the Kubernetes API CertificateSigningRequest for Approval

Using kubectl, submit the CSR request to the Kubernetes API. Here we can see that the CertificateSigningRequest: spec.groups entry is set to system:authenticated. This, upon approval, gives the Vault system pods authenticated access to the Kubernetes cluster environment.

kubectl create -f - <<-EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
 name: "$CSR_NAME"
spec:
 groups:
 - system:authenticated
 request: $( cat "$TMPDIR/server.csr" | base64 | tr -d '\r\n' )
 signerName: kubernetes.io/kubelet-serving
 usages:
 - digital signature
 - key encipherment
 - server auth
EOF

**CertificateSigningRequest Troubleshooting: server.csr file base64 decoding on Mac OSX**

You may have an issue rendering the server.csr file to base64 and may need to install the GNU coreutils and change the base64 decoding of the server.csr.

Install Mac OSX GNU Coreutils via homebrew:

brew install coreutils findutils gnu-tar gnu-sed gawk gnutls gnu-indent gnu-getopt grep

GNU Coreutils - base64 decoding:

Alter the server.csr request line in the Kubernetes CertificateSigningRequest as follows:

request: $( gbase64 -w 0 "$TMPDIR/server.csr" )

Approve the Kubernetes CSR Submittal and Verify

kubectl certificate approve "$CSR_NAME"

certificatesigningrequest.certificates.k8s.io/vault-csr approved

Verify the CSR was Approved and the TLS Certificate Issued

kubectl get csr "$CSR_NAME"

NAME      AGE  SIGNERNAME                    REQUESTOR        REQUESTEDDURATION  CONDITION
vault-csr 8s   kubernetes.io/kubelet-serving kubernetes-admin <none>             Approved,Issued

Obtain Kubernetes API Certificate Authority (CA) PEM Data

Next, we'll obtain the base64 PEM encoded Kubernetes CA data and place it into our $TMPDIR/ directory for use in a Kubernetes Secret.

kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > $TMPDIR/vault.ca

Create Kubernetes Secret for Vault TLS Certificate, Key, and CA

Finally, we'll store our Vault TLS Certificate, Key, and Kubernetes CA data in a Kubernetes Generic secret to use for reference in any future Consul or Vault Helm Override Values:

Note: Repeat this secret creation in any other Kubernetes namespaces that require the Vault TLS Certificate Data (i.e., Secret Volume Mounts / CA Certificate Recognition for Vault Secrets backend). See Kubernetes: Distribute Credentials Securely Using Secrets for more information.

kubectl create secret generic $SECRET_NAME \
    --namespace="$NAMESPACE" \
    --from-file=vault.key="$TMPDIR/vault.key" \
    --from-file=vault.crt="$TMPDIR/vault.crt" \
    --from-file=vault.ca="$TMPDIR/vault.ca"

References: