How to Pass a CA Certificate to the Vault Agent Injector from an External Vault Cluster

Avatar
Max Winslow
  • Updated

Introduction

Expected Outcome

A Vault Agent Injector deployed as a sidecar in Kubernetes environment can establish a TLS connection with an external Vault cluster (outside of the Kubernetes environment) and successfully retrieve secrets for application containers running in the same pod as the agent.

Prerequisites (if applicable)

  • Vault cluster with TLS running and unsealed outside of a Kubernetes environment (external)
  • The official Hashicorp Vault helm chart installed in your Kubernetes environment.
  • A Kubernetes pod that can be run with the Vault agent annotations (an example is provided in the procedure below)

Procedure

  • In the external vault cluster, let's enable a KV v2 secrets engine and add some data:

~ vault secrets enable -version=2 -path=secret kv 

~ vault kv put secret/devwebapp/config username='giraffe' password='salsa'

  • Now let's deploy a pod with the agent and application containers to retrieve this secret. This guide uses the steps and components described in this tutorial. The tutorial describes establishing a non-TLS-encrypted connection with the external Vault. Since TLS is enabled in the external Vault, let's watch how this breaks:

~ cat > pod-devwebapp-with-annotations.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
name: devwebapp-with-annotations
labels:
app: devwebapp-with-annotations
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: 'devweb-app'
vault.hashicorp.com/agent-inject-secret-credentials.txt: 'secret/data/devwebapp/config'
spec:
serviceAccountName: internal-app
containers:
- name: app
image: burtlo/devwebapp-ruby:k8s
EOF

~ kubectl apply --filename pod-devwebapp-with-annotations.yaml

  • After deploying, we see that the pod is stuck in an initializing state. If we inspect the logs of the vault-agent-init container within the pod we can see that the agent doesn't trust the certificate provided by the external Vault cluster.

~ kubectl get pods

NAME READY STATUS RESTARTS AGE
devwebapp-with-annotations 0/2 Init:0/1 0 3m41s
vault-agent-injector-b6b747d46-6bxcq 1/1 Running 0 23h

~ kubectl logs devwebapp-with-annotations -c vault-agent-init
...
2023-07-28T18:37:44.572Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://external-vault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=1s
2023-07-28T18:37:45.573Z [INFO] agent.auth.handler: authenticating
2023-07-28T18:37:45.579Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://external-vault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=1.63s

  • To resolve, we need to provide the deployed pod with the CA certificate of the external Vault cluster. First, let's create a Kubernetes secret that includes the CA certificate:

~ cat > tls-secret.yaml << EOF
---
kind: Secret
apiVersion: v1
metadata:
name: vault-tls-secret
stringData:
ca-bundle.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
type: Opaque
EOF

~ kubectl apply -f tls-secret.yaml
~ kubectl delete pod devwebapp-with-annotations
pod "devwebapp-with-annotations" deleted

~ cat > pod-devwebapp-with-annotations.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
name: devwebapp-with-annotations
labels:
app: devwebapp-with-annotations
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: 'devweb-app'
vault.hashicorp.com/agent-inject-secret-credentials.txt: 'secret/data/devwebapp/config'
vault.hashicorp.com/tls-secret: vault-tls-secret
vault.hashicorp.com/ca-cert: /vault/tls/ca-bundle.crt
spec:
serviceAccountName: internal-app
containers:
- name: app
image: burtlo/devwebapp-ruby:k8s
EOF

kubectl apply --filename pod-devwebapp-with-annotations.yaml
  • Now we can get the pods and see that it is up and running and the secret is successfully rendered from the external Vault
~ kubectl get pods

NAME READY STATUS RESTARTS AGE
devwebapp-with-annotations 2/2 Running 0 2m15s
vault-agent-injector-b6b747d46-6bxcq 1/1 Running 0 23h

~ kubectl exec -it devwebapp-with-annotations -- cat /vault/secrets/credentials.txt

data: map[password:salsa username:giraffe]

Additional Information

 

Articles in this section

See more

Related articles