Introduction
Expected Outcome
The Vault Agent Injector deployed as a sidecar in a Kubernetes environment can establish a TLS connection with an external Vault cluster (outside of the Kubernetes environment) and successfully retrieve secrets for application containers running in the same pod as the agent.
Prerequisites (if applicable)
- A certificate and key generated for the Vault cluster with the appropriate SANs
- A Vault cluster with TLS enabled running outside of a Kubernetes environment
- The official HashiCorp Vault Helm chart installed in a Kubernetes environment
- An application pod that can be run with the Vault Agent annotations
Procedure
# Run these commands on the Vault cluster
# Enable a kv secrets engine
$ vault secrets enable -path=secret kv
# Add a kv secret
$ vault kv put secret/devwebapp/config username='giraffe' password='salsa'
# Start Minikube
$ minikube start
# Create variable for Minikube host
$ EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')
# Create a service account
$ kubectl create sa internal-app
# Define a service and a corresponding endpoint configured to address the EXTERNAL_VAULT_ADDR
$ cat > external-vault.yaml <<EOF
---
apiVersion: v1
kind: Service
metadata:
name: external-vault
namespace: default
spec:
ports:
- protocol: TCP
port: 8200
---
apiVersion: v1
kind: Endpoints
metadata:
name: external-vault
subsets:
- addresses:
- ip: $EXTERNAL_VAULT_ADDR
ports:
- port: 8200
EOF
# Create service
$ kubectl apply --filename external-vault.yaml
# Add the HashiCorp Helm repository
$ helm repo add hashicorp https://helm.releases.hashicorp.com
# Update the repositories
$ helm repo update
# Install the latest version of the Vault running in external mode
$ helm install vault hashicorp/vault \
--set "global.externalVaultAddr=https://external-vault:8200"
# Define mountable secret (required for k8s version 1.24+)
$ cat > vault-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
name: vault-token-g955r
annotations:
kubernetes.io/service-account.name: vault
type: kubernetes.io/service-account-token
EOF
# Create secret
$ kubectl apply -f vault-secret.yaml
# Create a variable named VAULT_HELM_SECRET_NAME that stores the secret name
$ VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-token-")).name')
# Get the JSON web token (JWT) from the k8s secret
$ TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)
# Retrieve the Kubernetes CA certificate
$ KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
# Retrieve the Kubernetes host URL
$ KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
# Enable the Kubernetes authentication method
$ vault auth enable kubernetes
# Configure k8s authentication method
$ vault write auth/kubernetes/config \
token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert="$KUBE_CA_CERT" \
issuer="https://kubernetes.default.svc.cluster.local"
# Create policy that enables read capability to the secret
$ vault policy write devwebapp - <<EOF
path "secret/devwebapp/config" {
capabilities = ["read"]
}
EOF
# Create a k8s authentication method role
$ vault write auth/kubernetes/role/devweb-app \
bound_service_account_names=internal-app \
bound_service_account_namespaces=default \
policies=devwebapp \
ttl=24h
# Define a secret that includes CA certificate
$ cat > tls-secret.yaml << EOF
---
kind: Secret
apiVersion: v1
metadata:
name: vault-tls-secret
stringData:
ca-bundle.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
type: Opaque
EOF
# Create secret
$ kubectl apply -f tls-secret.yaml
# Define an application pod with the Vault Agent annotations
$ cat > pod-devwebapp-with-annotations.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
name: devwebapp-with-annotations
labels:
app: devwebapp-with-annotations
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: 'devweb-app'
vault.hashicorp.com/agent-inject-secret-credentials.txt: 'secret/devwebapp/config'
vault.hashicorp.com/tls-secret: vault-tls-secret
vault.hashicorp.com/ca-cert: /vault/tls/ca-bundle.crt
spec:
serviceAccountName: internal-app
containers:
- name: app
image: burtlo/devwebapp-ruby:k8s
EOF
# Create application pod
$ kubectl apply --filename pod-devwebapp-with-annotations.yaml
# Display the secrets written to the file /vault/secrets/secret-credentials.txt on the devwebapp-with-annotations pod
$ kubectl exec -it devwebapp-with-annotations -c app -- cat /vault/secrets/credentials.txt
- Note the annotations for the application pod vault.hashicorp.com/tls-secret and vault.hashicorp.com/ca-cert
- The application pod is running and the secret is successfully rendered from the external Vault cluster
Additional Information
- https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-tls
- https://discuss.hashicorp.com/t/passing-a-ca-certificate-to-injected-vault-agent-and-vault-agent-init-containers/7795