Introduction
In a Consul Helm chart configuration that does not use Vault, webhook-cert-manager ensures that a valid certificate is updated to the mutatingwebhookconfiguration of either the controller or connect inject to ensure that Kubernetes can communicate with each of these services.
Please refer to the another article covering how webhook-cert-manager is being used to issue these certificates.
When Vault is configured as the controller and connected inject Webhook Certificate Provider on Kubernetes:
-
webhook-cert-manager is no longer deployed to the cluster.
-
Controller and connect inject each gets their webhook certificates from their own Vault PKI mount via the injected Vault Agent.
-
Controller and connect inject each need to be configured with their own Vault Role that has the necessary permissions to receive certificates from its respective PKI mount.
-
Controller and connect inject each locally updates its own mutatingwebhookconfiguration so that Kubernetes can relay events.
-
Vault manages certificate rotation and rotates certificates to each webhook.
Lab Setup
To keep the setup minimalistic, simply run a vault agent locally in dev mode, and create a kind cluster locally with 3 nodes to run the consul-k8s workload.
user1@server-GJ9F6YC4M5 ~ % vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200
==> Vault server configuration:
Api Address: http://0.0.0.0:8200
Cgo: disabled
Cluster Address: https://0.0.0.0:8201
Environment Variables: COLORFGBG, COLORTERM, COMMAND_MODE, GODEBUG, HOME, HOMEBREW_CELLAR, HOMEBREW_PREFIX, HOMEBREW_REPOSITORY, INFOPATH, ITERM_PROFILE, ITERM_SESSION_ID, LC_CTYPE, LC_TERMINAL, LC_TERMINAL_VERSION, LOGNAME, LaunchInstanceID, MANPATH, OLDPWD, PATH, PWD, SECURITYSESSIONID, SHELL, SHLVL, SSH_AUTH_SOCK, TERM, TERM_PROGRAM, TERM_PROGRAM_VERSION, TERM_SESSION_ID, TMPDIR, USER, XPC_FLAGS, XPC_SERVICE_NAME, _, __CFBundleIdentifier, __CF_USER_TEXT_ENCODING
Go Version: go1.20.3
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level:
Mlock: supported: false, enabled: false
Recovery Mode: false
Storage: inmem
Version: Vault v1.13.2, built 2023-04-25T13:02:50Z
Version Sha: b9b773f1628260423e6cc9745531fd903cae853f
....Set the VAULT_ADDR environment variable to access the local running vault cluster, and then do `vault login ...` to perform further actions on it.
user1@server-GJ9F6YC4M5 ~ % export VAULT_ADDR=http://0.0.0.0:8200
user1@server-GJ9F6YC4M5 ~ % vault login root
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token root
token_accessor oxwFwLt6VjdueMOKKuEgtHxe
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
A service bound to all networks on the host, as you configured Vault, is addressable by pods in kind cluster by sending requests to the gateway address of the Kubernetes cluster.
From docker-worker2 or any other container, do an exec:-
# dig +short host.docker.internal
192.168.65.254
#Note:- The host has a changing IP address (or none if you have no network access). We recommend that you connect to the special DNS name host.docker.internal which resolves to the internal IP address used by the host. host.docker.internal.
Create a variable named EXTERNAL_VAULT_ADDR to capture the kind gateway address.
Later, we install vault helm chart to run vault-agent-injector pod which would be pointing to external-vault server using EXTERNAL_VAULT_ADDR
user1@server-GJ9F6YC4M5 ~ % export EXTERNAL_VAULT_ADDR=192.168.65.254
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ % helm install vault hashicorp/vault \
--set "global.externalVaultAddr=http://$EXTERNAL_VAULT_ADDR:8200"
NAME: vault
LAST DEPLOYED: Fri Apr 19 10:57:18 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Vault!
Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:
https://developer.hashicorp.com/vault/docs
Your release is named vault. To learn more about the release, try:
$ helm status vault
$ helm get manifest vault
user1@server-GJ9F6YC4M5 ~ % alias k=kubectl
user1@server-GJ9F6YC4M5 ~ % k get po -w
NAME READY STATUS RESTARTS AGE
vault-agent-injector-bdcbb8645-xfw4c 0/1 ContainerCreating 0 5s
vault-agent-injector-bdcbb8645-xfw4c 1/1 Running 0 16s
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ % k logs vault-agent-injector-bdcbb8645-xfw4c -f
2024-04-19T05:27:30.048Z [INFO] handler.auto-tls: Generated CA
Listening on ":8080"...
2024-04-19T05:27:30.052Z [INFO] handler: Starting handler..
2024-04-19T05:27:30.148Z [INFO] handler.certwatcher: Webhooks changed. Updating certs...
2024-04-19T05:27:30.148Z [WARN] handler.certwatcher: Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...
2024-04-19T05:27:30.148Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2024-04-19T05:27:30.156Z [INFO] handler.certwatcher: Webhooks changed. Updating certs...
^C
user1@server-GJ9F6YC4M5 ~ % k get sa
NAME SECRETS AGE
default 0 7d23h
vault 0 2m26s
vault-agent-injector 0 2m26s
user1@server-GJ9F6YC4M5 ~ % k get secrets
NAME TYPE DATA AGE
sh.helm.release.v1.vault.v1 helm.sh/release.v1 1 2m31s
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ %
User needs to generate tokens for the different nodes and servers to securely configure Consul. In order to generate these tokens, what is need is a key for gossip encryption, a Consul Enterprise license, TLS certificates for the Consul servers and ACL policies
Since you are using Vault as secrets management for your Consul datacenter, all the secrets will be stored inside the Vault.
Vault provides a PKI secrets engine that you can use to generate TLS certificates. This engine is used to configure CAs for TLS encryption for servers and service mesh leaf certificates for services.
user1@server-GJ9F6YC4M5 ~ % vault secrets enable -path=consul kv-v2
Success! Enabled the kv-v2 secrets engine at: consul/
user1@server-GJ9F6YC4M5 ~ % vault kv put consul/secret/enterpriselicense key="$(cat ./consul.hclic)"
user1@server-GJ9F6YC4M5 ~ % vault kv put consul/secret/gossip gossip="$(consul keygen)"
====== Secret Path ======
consul/data/secret/gossip
======= Metadata =======
Key Value
--- -----
created_time 2024-04-19T05:46:52.569865Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
user1@server-GJ9F6YC4M5 ~ % vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ % vault secrets tune -max-lease-ttl=87600h pki
Success! Tuned the secrets engine at: pki/
user1@server-GJ9F6YC4M5 ~ % vault write -field=certificate pki/root/generate/internal \
common_name="dc1.consul" \
ttl=87600h | tee consul_ca.crt
-----BEGIN CERTIFICATE-----
MIIDMjCCAhqgAwIBAgIUa/ARLqPsmFRPXer5zTIMwYMdhTQwDQYJKoZIhvcNAQEL
BQAwFTETMBEGA1UEAxMKZGMxLmNvbnN1bDAeFw0yNDA0MTkwNjA3MDNaFw0zNDA0
MTcwNjA3MzNaMBUxEzARBgNVBAMTCmRjMS5jb25zdWwwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQCdgnyF5eSa+hX1ghT6dUBbwOktLTAj6gs4jswedo6k
WWN+QyzSCCadRXGNaK8I5iGaH80o9/cRGDiXVxbVbHzfVbB89i38+uHALOwcFozf
....
-----END CERTIFICATE-----%
user1@server-GJ9F6YC4M5 kind % vault write pki/roles/consul-server \
allowed_domains="consul-server.default,consul-server.default.svc,*.consul-server.default,*.consul-server.default.svc,dc1.consul,consul-server,consul-server.consul,consul-server.consul.svc" \
allow_subdomains=true \
allow_bare_domains=true \
allow_localhost=true \
generate_lease=true \
max_ttl="768h"
WARNING! The following warnings were returned from Vault:
* it is encouraged to disable generate_lease and rely on PKI's native
capabilities when possible; this option can cause Vault-wide issues with
large numbers of issued certificates
Key Value
--- -----
allow_any_name false
allow_bare_domains true
allow_glob_domains false
allow_ip_sans true
allow_localhost true
allow_subdomains true
allow_token_displayname false
allow_wildcard_certificates true
allowed_domains [consul-server.default consul-server.default.svc *.consul-server.default *.consul-server.default.svc dc1.consul consul-server consul-server.consul consul-server.consul.svc]
...
user1@server-GJ9F6YC4M5 ~ % vault secrets enable -path connect-root pki
Success! Enabled the pki secrets engine at: connect-root/
user1@server-GJ9F6YC4M5 ~ %
Now configure the Kubernetes authentication method and define the different policies to grant access to the resources. Then define the associations between Kubernetes service accounts and Vault policies.
Create vault serviceAccount’s secrets (due to changes after K8s 1.22 version), and its respective cluster-role-bindings to grant system:auth-delegator role.
user1@server-GJ9F6YC4M5 ~ % cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: Secret
metadata:
name: vault
namespace: default
annotations:
kubernetes.io/service-account.name: vault
type: kubernetes.io/service-account-token
EOF
secret/vault created
user1@server-GJ9F6YC4M5 ~ % cat <<EOF | kubectl create -f -
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault
namespace: default
EOF
clusterrolebinding.rbac.authorization.k8s.io/role-tokenreview-binding created
user1@server-GJ9F6YC4M5 ~ % k get secrets
NAME TYPE DATA AGE
sh.helm.release.v1.vault.v1 helm.sh/release.v1 1 46m
vault kubernetes.io/service-account-token 3 111s
user1@server-GJ9F6YC4M5 ~ % vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
user1@server-GJ9F6YC4M5 ~ % export TOKEN_REVIEW_JWT=$(kubectl get secret vault -o go-template='{{ .data.token }}' | base64 --decode)
user1@server-GJ9F6YC4M5 ~ % echo $TOKEN_REVIEW_JWT
eyJhbGciOiJSUzI1NiIsImtpZCI6IkhGcG5KaWFsUnhYLWFsbDE1Rl94MnZvQzJQdz...
user1@server-GJ9F6YC4M5 ~ % export KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)
user1@server-GJ9F6YC4M5 ~ % echo $KUBE_CA_CERT
-----BEGIN CERTIFICATE-----
MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTI0MDQxMTA2MDgyM1oXDTM0MDQwOTA2MDgyM1owFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM9l
s0cc/9YMMfuiud1VHWbwkqm7upeyuqTcI/9Ffjz/XhgQGK1UEXRqzCOG++cCGGPj
+YZFbkBlPegleaH14V6HDCWiJbSgpau4gb/WLF+HXONslkrdU6oUiyvEU9mS3+4B
IB8HiKwggAsT/VUu3kvl9EbPTdmTJLuKJU4wTpzSbO1TkykYKXbPb7OO7MCyyqZF
...
-----END CERTIFICATE-----
user1@server-GJ9F6YC4M5 ~ % export KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
user1@server-GJ9F6YC4M5 ~ % echo $KUBE_HOST
https://127.0.0.1:57588
user1@server-GJ9F6YC4M5 ~ % vault write auth/kubernetes/config token_reviewer_jwt="$TOKEN_REVIEW_JWT" kubernetes_host="$KUBE_HOST" kubernetes_ca_cert="$KUBE_CA_CERT"
Success! Data written to: auth/kubernetes/config
user1@server-GJ9F6YC4M5 ~ % vault read auth/kubernetes/config
Key Value
--- -----
disable_iss_validation true
disable_local_ca_jwt false
issuer n/a
kubernetes_ca_cert -----BEGIN CERTIFICATE-----
MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTI0MDQxMTA2MDgyM1oXDTM0MDQwOTA2MDgyM1owFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM9l
s0cc/9YMMfuiud1VHWbwkqm7upeyuqTcI/9Ffjz/XhgQGK1UEXRqzCOG++cCGGPj
+YZFbkBlPegleaH14V6HDCWiJbSgpau4gb/WLF+HXONslkrdU6oUiyvEU9mS3+4B
...
-----END CERTIFICATE-----
kubernetes_host https://127.0.0.1:57588
pem_keys []
We must define the different Vault policies that will let the Consul agents generate or retrieve the different secrets.
Create Vault policies to grant access to:
-
The gossip encryption key.
-
Consul server policy.
-
Consul CA access policy.
-
Consul service mesh CA policy.
-
Consul enterprise license[In case of Consul Enterprise version]
user1@server-GJ9F6YC4M5 ~ % vault policy write gossip-policy - <<EOF
path "consul/data/secret/gossip" {
capabilities = ["read"]
}
EOF
Success! Uploaded policy: gossip-policy
user1@server-GJ9F6YC4M5 ~ % vault policy write enterpriselicense-policy - <<EOF
path "consul/data/secret/enterpriselicense" {
capabilities = ["read"]
}
EOF
Success! Uploaded policy: enterpriselicense-policy
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ % vault policy write consul-server - <<EOF
path "kv/data/consul-server"
{
capabilities = ["read"]
}
path "pki/issue/consul-server"
{
capabilities = ["read","update"]
}
path "pki/cert/ca"
{
capabilities = ["read"]
}
EOF
Success! Uploaded policy: consul-server
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ % vault policy write ca-policy - <<EOF
path "pki/cert/ca" {
capabilities = ["read"]
}
EOF
Success! Uploaded policy: ca-policy
user1@server-GJ9F6YC4M5 ~ % vault policy write connect - <<EOF
path "/sys/mounts/connect-root" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "/sys/mounts/connect-intermediate-dc1" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "/sys/mounts/connect-intermediate-dc1/tune" {
capabilities = [ "update" ]
}
path "/connect-root/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "/connect-intermediate-dc1/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "auth/token/renew-self" {
capabilities = [ "update" ]
}
path "auth/token/lookup-self" {
capabilities = [ "read" ]
}
EOF
Success! Uploaded policy: connect
user1@server-GJ9F6YC4M5 ~ %
Also, create Vault roles to associate the necessary policies to:
-
Consul server agents.
-
Consul client agents.
-
Consul CA certificate access.
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ % vault write auth/kubernetes/role/consul-server \
bound_service_account_names=consul-server \
bound_service_account_namespaces="*" \
policies="gossip-policy,consul-server,connect,enterpriselicense-policy" \
ttl=24h
Success! Data written to: auth/kubernetes/role/consul-server
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ % vault write auth/kubernetes/role/consul-client \
bound_service_account_names=consul-client \
bound_service_account_namespaces="*" \
policies="gossip-policy,ca-policy,enterpriselicense-policy" \
ttl=24h
Success! Data written to: auth/kubernetes/role/consul-client
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ %
user1@server-GJ9F6YC4M5 ~ % vault write auth/kubernetes/role/consul-ca \
bound_service_account_names="*" \
bound_service_account_namespaces="*" \
policies=ca-policy \
ttl=1h
Success! Data written to: auth/kubernetes/role/consul-ca
user1@server-GJ9F6YC4M5 ~ %
The way secret engines, root certificate, policies, rules and K8s-auth-roles (to bound a serviceAccount in a particular namespace to correct policies to get privileges) for consul-servers, consul-client and consul-ca is created, same way create secret engines, root certificate, policies, rules and K8s-auth-roles for controller and connect-injector which are going to fetch respective secrets stored over vault.
user1@server-GJ9F6YC4M5 kind % vault secrets enable -path=controller pki
Success! Enabled the pki secrets engine at: controller/
user1@server-GJ9F6YC4M5 kind % vault secrets enable -path=connect-inject pki
Success! Enabled the pki secrets engine at: connect-inject/
user1@server-GJ9F6YC4M5 kind %
user1@server-GJ9F6YC4M5 kind % vault secrets tune -max-lease-ttl=87600h controller
Success! Tuned the secrets engine at: controller/
user1@server-GJ9F6YC4M5 kind % vault secrets tune -max-lease-ttl=87600h connect-inject
Success! Tuned the secrets engine at: connect-inject/
user1@server-GJ9F6YC4M5 kind % vault write -field=certificate controller/root/generate/internal \
common_name="consul-controller-webhook" \
ttl=87600h
-----BEGIN CERTIFICATE-----
MIIDYTCCAkmgAwIBAgIUVpQ0B9cULXZ/nEB+eMJ9xb4FVAQwDQYJKoZIhvcNAQEL
BQAwJDEiMCAGA1UEAxMZY29uc3VsLWNvbnRyb2xsZXItd2ViaG9vazAeFw0yNDA0
MTkwNjUxMDhaFw0zNDA0MTcwNjUxMzhaMCQxIjAgBgNVBAMTGWNvbnN1bC1jb250
cm9sbGVyLXdlYmhvb2swggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8
m/MtaYZ8O0OQDrNm7Tvu2JeQdWpUKOVfhzRcJcTUOLMKlOUXWFr9tnu3y3q73hhU
fIuDQCuZ/F3fy567CvIAPfu2LMN4y3oPHAIPwhwCSOcVVyoU3tSGEstVCcfYdNpO
ZxFWQrDvRbM6MTNdy7l6dOTmfEncVEkI+z7JzIJ+XcTvyyd/8dqIVCJV9MjF1fHS
RqTijzzXrRQt0GxosnvcftWhesuwNgX0hAZFo2K6+Fg3voPm/g6KRkeCp13z3Uwd
54WUDMmxgXc/9chbyfl45ZSnwNLnT8kYToLG7MWcSn75L9U5xbMr/znMMsP2+yHw
...
-----END CERTIFICATE-----
user1@server-GJ9F6YC4M5 kind % vault write -field=certificate connect-inject/root/generate/internal \
common_name="consul-connect-injector" \
ttl=87600h
-----BEGIN CERTIFICATE-----
MIIDWzCCAkOgAwIBAgIUQPud82Rld0cAhgY+HRhnp6vn1SAwDQYJKoZIhvcNAQEL
BQAwIjEgMB4GA1UEAxMXY29uc3VsLWNvbm5lY3QtaW5qZWN0b3IwHhcNMjQwNDE5
MDY1MTI1WhcNMzQwNDE3MDY1MTU1WjAiMSAwHgYDVQQDExdjb25zdWwtY29ubmVj
dC1pbmplY3RvcjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK7zz+xo
Y2HhDNNF5tI/SoETo59VJpTS97SSIWbXozmjZSVh1Fp6ONYHIQZ9EgK7lF1qObvU
F7IyUWd1glu/jsYuxoubTIXaVdr7jSRQzZrMOmusSo0Y97WtnCB+j9TZ6PSOmQ9K
VsNjmdjF6VJEY/h1UwXdeCbAxAVRAU/ir8hHdZ+dtUeKx/yel22ynUdQwjYvt8vd
oZfflsMbdzcDRe3qm0rE3ElTSvPCcTpSCcnAmvCZ+InsiwKrBkIyQ1l79j3NOjCL
ivrspchClqojY4D6gQ3e3rp7vpadOYh/zN1j2+Z5RoXNMtNG71oLJdDC1b8edpgf
...
-----END CERTIFICATE-----
user1@server-GJ9F6YC4M5 kind % vault policy write controller-tls-policy - <<EOF
path "controller/issue/controller-role" {
capabilities = ["create", "update"]
}
EOF
Success! Uploaded policy: controller-tls-policy
user1@server-GJ9F6YC4M5 kind % vault policy write connect-inject-policy - <<EOF
path "connect-inject/issue/connect-inject-role" {
capabilities = ["create", "update"]
}
EOF
Success! Uploaded policy: connect-inject-policy
user1@server-GJ9F6YC4M5 kind % vault policy write controller-ca-policy - <<EOF
path "controller/cert/ca" {
capabilities = ["read"]
}
path "controller/issue/connect-inject-role"
{
capabilities = ["read","update"]
}
EOF
Success! Uploaded policy: controller-ca-policy
user1@server-GJ9F6YC4M5 kind % vault policy write connect-inject-ca-policy - <<EOF
path "connect-inject/cert/ca" {
capabilities = ["read"]
}
path "connect-inject/issue/connect-inject-role"
{
capabilities = ["read","update"]
}
EOF
Success! Uploaded policy: connect-inject-ca-policy
user1@server-GJ9F6YC4M5 kind % vault write connect-inject/roles/connect-inject-role \
allowed_domains="consul-connect-injector,consul-connect-injector.default,consul-connect-injector.default.svc,consul-connect-injector.default.svc.cluster.local" \
allow_subdomains=true \
allow_bare_domains=true \
allow_localhost=true \
generate_lease=true \
max_ttl="720h"
WARNING! The following warnings were returned from Vault:
* it is encouraged to disable generate_lease and rely on PKI's native
capabilities when possible; this option can cause Vault-wide issues with
large numbers of issued certificates
Key Value
--- -----
allow_any_name false
allow_bare_domains true
allow_glob_domains false
allow_ip_sans true
allow_localhost true
allow_subdomains true
allow_token_displayname false
allow_wildcard_certificates true
allowed_domains [consul-connect-injector consul-connect-injector.default consul-connect-injector.default.svc consul-connect-injector.default.svc.cluster.local]
...
user1@server-GJ9F6YC4M5 kind % vault write controller/roles/controller-role \
allowed_domains="consul-controller-webhook,consul-controller-webhook.default,consul-controller-webhook.default.svc,consul-controller-webhook.default.svc.cluster.local" \
allow_subdomains=true \
allow_bare_domains=true \
allow_localhost=true \
generate_lease=true \
max_ttl="720h"
WARNING! The following warnings were returned from Vault:
* it is encouraged to disable generate_lease and rely on PKI's native
capabilities when possible; this option can cause Vault-wide issues with
large numbers of issued certificates
Key Value
--- -----
allow_any_name false
allow_bare_domains true
allow_glob_domains false
allow_ip_sans true
allow_localhost true
allow_subdomains true
allow_token_displayname false
allow_wildcard_certificates true
allowed_domains [consul-controller-webhook consul-controller-webhook.default consul-controller-webhook.default.svc consul-controller-webhook.default.svc.cluster.local]
...
user1@server-GJ9F6YC4M5 kind % vault write auth/kubernetes/role/controller-role \
bound_service_account_names="*" \
bound_service_account_namespaces="*" \
policies=controller-ca-policy \
ttl=1h
Success! Data written to: auth/kubernetes/role/controller-role
user1@server-GJ9F6YC4M5 kind % vault write auth/kubernetes/role/connect-inject-role \
bound_service_account_names="*" \
bound_service_account_namespaces="*" \
policies=connect-inject-ca-policy \
ttl=1h
Success! Data written to: auth/kubernetes/role/connect-inject-role
Now Vault is configured and the user has to configure the Consul Helm chart to use vault as webhook certificate provider for connect-injector and controller components.
However, for this exercise controller is not used as it has been merged into connect-injector from consul-k8s 1.0.0 version. (Ref. https://github.com/hashicorp/consul-k8s/pull/1697 )
user1@server-GJ9F6YC4M5 kind % cat values_with_vault_provider_for_webhook.yaml
global:
datacenter: "dc1"
name: consul
domain: consul
image: hashicorp/consul-enterprise:1.18.1-ent
secretsBackend:
vault:
enabled: true
consulServerRole: consul-server
consulClientRole: consul-client
consulCARole: consul-ca
controllerRole: "controller-role"
connectInjectRole: "connect-inject-role"
connectInject:
caCert:
secretName: "connect-inject/cert/ca"
tlsCert:
secretName: "connect-inject/issue/connect-inject-role"
connectCA:
address: $EXTERNAL_VAULT_ADDR
rootPKIPath: connect-root/
intermediatePKIPath: connect-intermediate-dc1/
#additionalConfig: "{\"connect\": [{ \"ca_config\": [{ \"namespace\": \"admin\"}]}]}" # We would need it in case vault is running in some other custom namespace
#agentAnnotations: |
#"vault.hashicorp.com/namespace": "admin" #Like in HCP Vault, namespace would be `admin`
enterpriseLicense:
secretName: 'consul/data/secret/enterpriselicense'
secretKey: 'key'
tls:
enabled: true
enableAutoEncrypt: true
caCert:
secretName: "pki/cert/ca"
federation:
enabled: false
createFederationSecret: false
acls:
manageSystemACLs: false
gossipEncryption:
secretName: consul/data/secret/gossip
secretKey: gossip
enableConsulNamespaces: true
server:
replicas: 3
serverCert:
secretName: "pki/issue/consul-server"
connectInject:
enabled: true
$ helm install consul hashicorp/consul --values values_with_vault_provider_for_webhook.yaml --wait --debug
...
user1@server-GJ9F6YC4M5 ~ % k get po
NAME READY STATUS RESTARTS AGE
consul-connect-injector-57b7c994b8-jcl6c 2/2 Running 0 5m52s
consul-server-0 2/2 Running 0 5m52s
consul-server-1 2/2 Running 0 5m52s
consul-server-2 2/2 Running 0 5m52s
vault-agent-injector-bdcbb8645-xfw4c 1/1 Running 0 5h31m
user1@server-GJ9F6YC4M5 ~ % k get secrets
NAME TYPE DATA AGE
sh.helm.release.v1.consul.v1 helm.sh/release.v1 1 6m21s
sh.helm.release.v1.vault.v1 helm.sh/release.v1 1 5h31m
vault kubernetes.io/service-account-token 3 4h46m
Upon installation, mutatingwebhookconfiguration object was created and its caBundle values is updated via vault webhook certificate manager.
user1@server-GJ9F6YC4M5 ~ % k get mutatingwebhookconfiguration
NAME WEBHOOKS AGE
consul-connect-injector 13 17m
vault-agent-injector-cfg 1 5h42m
user1@server-GJ9F6YC4M5 ~ % k describe mutatingwebhookconfiguration vault-agent-injector-cfg
Name: vault-agent-injector-cfg
Namespace:
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=vault-agent-injector
Annotations: meta.helm.sh/release-name: vault
meta.helm.sh/release-namespace: default
API Version: admissionregistration.k8s.io/v1
Kind: MutatingWebhookConfiguration
Metadata:
Creation Timestamp: 2024-04-19T05:27:19Z
Generation: 2
Resource Version: 451784
UID: 1f4e035a-6c31-4997-a005-100cbc03c8e5
Webhooks:
Admission Review Versions:
v1
v1beta1
Client Config:
Ca Bundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNTRENDQWU2Z0F3SUJBZ0lVTlBnenloeDFaNVN3TV....
Service:
Name: vault-agent-injector-svc
Namespace: default
Path: /mutate
Port: 443
Failure Policy: Ignore
Match Policy: Exact
Name: vault.hashicorp.com
Namespace Selector:
Object Selector:
Match Expressions:
Key: app.kubernetes.io/name
Operator: NotIn
Values:
vault-agent-injector
Reinvocation Policy: Never
Rules:
API Groups:
API Versions:
v1
Operations:
CREATE
UPDATE
Resources:
pods
Scope: *
Side Effects: None
Timeout Seconds: 30
Events: <none>
user1@server-GJ9F6YC4M5 ~ % echo "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNTRENDQWU2Z0F3SUJBZ0lVTlBnenl..." | base64 -d | openssl x509 -text -noout
Warning: Reading certificate from stdin since no -in or -new option is given
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
34:f8:33:ca:1c:75:67:94:b0:31:2a:d5:1d:3b:f5:11:f0:9b:bf:8f
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN=Agent Inject CA
Validity
Not Before: Apr 19 05:26:30 2024 GMT
Not After : Apr 17 05:27:30 2034 GMT
Subject: CN=Agent Inject CA
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:cd:ba:4a:09:85:b9:bb:f2:34:bc:66:db:e7:1e:
89:9d:6c:e0:a5:63:6f:55:e3:d5:6b:3e:ad:e4:b6:
1c:c8:34:ef:a2:0c:b0:e9:ba:3e:95:3a:a5:db:2a:
d9:ab:5c:5a:8d:08:26:58:19:f1:f5:e4:cf:6f:c3:
33:90:57:10:a5
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Certificate Sign
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
37:66:3A:31:33:3A:66:35:3A:34:66:3A:63:39:3A:36:38:3A:30:32:3A:66:34:3A:66:30:3A:64:38:3A:33:33:3A:61:32:3A:38:62:3A:37:63:3A:35:33:3A:30:30:3A:31:35:3A:65:33:3A:33:37:3A:61:33:3A:39:61:3A:62:36:3A:37:65:3A:65:39:3A:39:32:3A:39:39:3A:35:34:3A:38:39:3A:33:36:3A:38:63:3A:65:32:3A:33:30
X509v3 Authority Key Identifier:
37:66:3A:31:33:3A:66:35:3A:34:66:3A:63:39:3A:36:38:3A:30:32:3A:66:34:3A:66:30:3A:64:38:3A:33:33:3A:61:32:3A:38:62:3A:37:63:3A:35:33:3A:30:30:3A:31:35:3A:65:33:3A:33:37:3A:61:33:3A:39:61:3A:62:36:3A:37:65:3A:65:39:3A:39:32:3A:39:39:3A:35:34:3A:38:39:3A:33:36:3A:38:63:3A:65:32:3A:33:30
Signature Algorithm: ecdsa-with-SHA256
Signature Value:
30:45:02:21:00:93:db:d1:91:98:78:11:f4:f2:f8:9b:f8:e5:
4f:14:be:7d:83:85:d6:2a:d3:ab:f8:19:5c:70:f0:62:da:10:
d3:02:20:54:80:43:cf:e9:9c:bc:c9:0f:8b:75:23:68:b4:2e:
04:98:ab:47:7e:86:55:ed:f0:b1:32:27:b6:7d:05:b7:d2
user1@server-GJ9F6YC4M5 ~ % k describe mutatingwebhookconfiguration consul-connect-injector
Name: consul-connect-injector
Namespace:
Labels: app=consul
app.kubernetes.io/managed-by=Helm
chart=consul-helm
component=connect-injector
heritage=Helm
release=consul
Annotations: meta.helm.sh/release-name: consul
meta.helm.sh/release-namespace: default
API Version: admissionregistration.k8s.io/v1
Kind: MutatingWebhookConfiguration
Metadata:
Creation Timestamp: 2024-04-19T10:52:30Z
Generation: 2
Resource Version: 488015
UID: f25d8f89-6517-4658-9c7f-3b11454a22d1
Webhooks:
Admission Review Versions:
v1beta1
v1
Client Config:
Ca Bundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURXek....
Service:
Name: consul-connect-injector
Namespace: default
Path: /mutate-v1alpha1-proxydefaults
Port: 443
Failure Policy: Fail
Match Policy: Equivalent
Name: mutate-proxydefaults.consul.hashicorp.com
Namespace Selector:
Object Selector:
Reinvocation Policy: Never
Rules:
API Groups:
consul.hashicorp.com
API Versions:
v1alpha1
Operations:
CREATE
UPDATE
Resources:
proxydefaults
Scope: *
Side Effects: None
Timeout Seconds: 10
Admission Review Versions:
v1beta1
v1
....
user1@server-GJ9F6YC4M5 ~ % echo "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1..." | base64 -d| openssl x509 -text -noout
Warning: Reading certificate from stdin since no -in or -new option is given
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
40:fb:9d:f3:64:65:77:47:00:86:06:3e:1d:18:67:a7:ab:e7:d5:20
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=consul-connect-injector
Validity
Not Before: Apr 19 06:51:25 2024 GMT
Not After : Apr 17 06:51:55 2034 GMT
Subject: CN=consul-connect-injector
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:ae:f3:cf:ec:68:63:61:e1:0c:d3:45:e6:d2:3f:
4a:81:13:a3:9f:55:26:94:d2:f7:b4:92:21:66:d7:
a3:39:a3:65:25:61:d4:5a:7a:38:d6:07:21:06:7d:
12:02:bb:94:5d:6a:39:bb:d4:17:b2:32:51:67:75:
82:5b:bf:8e:c6:2e:c6:8b:9b:4c:85:da:55:da:fb:
8d:24:50:cd:9a:cc:3a:6b:ac:4a:8d:18:f7:b5:ad:
9c:20:7e:8f:d4:d9:e8:f4:8e:99:0f:4a:56:c3:63:
99:d8:c5:e9:52:44:63:f8:75:53:05:dd:78:26:c0:
c4:05:51:01:4f:e2:af:c8:47:75:9f:9d:b5:47:8a:
c7:fc:9e:97:6d:b2:9d:47:50:c2:36:2f:b7:cb:dd:
a1:97:df:96:c3:1b:77:37:03:45:ed:ea:9b:4a:c4:
dc:49:53:4a:f3:c2:71:3a:52:09:c9:c0:9a:f0:99:
f8:89:ec:8b:02:ab:06:42:32:43:59:7b:f6:3d:cd:
3a:30:8b:8a:fa:ec:a5:c8:42:96:aa:23:63:80:fa:
81:0d:de:de:ba:7b:be:96:9d:39:88:7f:cc:dd:63:
db:e6:79:46:85:cd:32:d3:46:ef:5a:0b:25:d0:c2:
d5:bf:1e:76:98:1f:13:8e:65:15:d4:bd:6a:a5:a3:
9e:c1
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
79:6C:AF:F8:49:21:D6:03:29:DD:34:69:DB:69:61:EA:73:4E:C0:3D
X509v3 Authority Key Identifier:
79:6C:AF:F8:49:21:D6:03:29:DD:34:69:DB:69:61:EA:73:4E:C0:3D
X509v3 Subject Alternative Name:
DNS:consul-connect-injector
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
10:39:10:08:ad:9b:5d:8f:d2:e3:68:0f:35:04:5c:f7:38:b0:
31:5f:df:90:59:18:16:61:ee:ee:25:9b:f7:50:04:e0:cd:14:
62:70:10:a6:4c:4c:a4:8a:90:3b:f8:9e:02:46:44:44:36:80:
01:dc:eb:a4:66:56:4d:47:f1:47:69:1c:a6:b4:f2:04:3a:3f:
93:0f:8b:ee:e3:ee:1d:4a:79:ea:2a:9b:0b:14:bb:13:96:5f:
8d:5c:60:29:3a:21:c4:e0:ad:44:53:a3:52:35:b1:e4:7a:42:
8c:a0:ca:b5:98:c4:91:58:77:7e:6b:db:ca:44:c9:4f:0b:ed:
e7:9f:69:dc:49:2d:bb:4b:13:68:4a:4e:a6:a8:c1:96:bb:bd:
a2:b8:1d:7d:81:7e:b0:bf:0e:94:11:1a:dd:5f:ec:24:b8:4a:
28:cc:f8:88:37:46:86:30:64:cf:a2:e0:55:73:0c:a8:60:ab:
5b:53:38:3b:d8:22:79:00:43:32:92:6e:46:8c:8b:80:81:03:
51:98:4f:de:4d:d8:9e:77:51:d9:a4:21:67:30:5f:b5:38:84:
cc:98:1b:86:81:4e:da:0d:5f:e3:29:97:90:5a:59:30:86:88:
9f:17:23:5c:d3:c9:c8:48:f4:ff:8b:5c:76:68:2b:48:44:b1:
f8:3e:ef:d8
user1@server-GJ9F6YC4M5 ~ %To validate how these certificates etc. being pulled into connect-injector let’s us exec into the connect-injector pod (please refer to the highlighted part).
user1@server-GJ9F6YC4M5 ~ % k describe po consul-connect-injector-57b7c994b8-jcl6c
Name: consul-connect-injector-57b7c994b8-jcl6c
Namespace: default
Priority: 0
Service Account: consul-connect-injector
Node: dc1-worker/172.18.0.4
Start Time: Fri, 19 Apr 2024 16:22:30 +0530
Labels: app=consul
chart=consul-helm
component=connect-injector
pod-template-hash=57b7c994b8
release=consul
Annotations: consul.hashicorp.com/connect-inject: false
consul.hashicorp.com/mesh-inject: false
vault.hashicorp.com/agent-init-first: true
vault.hashicorp.com/agent-inject: true
vault.hashicorp.com/agent-inject-secret-ca.crt: connect-inject/cert/ca
vault.hashicorp.com/agent-inject-secret-serverca.crt: pki/cert/ca
vault.hashicorp.com/agent-inject-secret-tls.crt: connect-inject/issue/connect-inject-role
vault.hashicorp.com/agent-inject-secret-tls.key: connect-inject/issue/connect-inject-role
vault.hashicorp.com/agent-inject-status: injected
vault.hashicorp.com/agent-inject-template-ca.crt:
{{- with secret "connect-inject/cert/ca" -}}
{{- .Data.certificate -}}
{{- end -}}
vault.hashicorp.com/agent-inject-template-serverca.crt:
{{- with secret "pki/cert/ca" -}}
{{- .Data.certificate -}}
{{- end -}}
vault.hashicorp.com/agent-inject-template-tls.crt:
{{- with secret "connect-inject/issue/connect-inject-role" "common_name=consul-connect-injector"
"alt_names=consul-connect-injector,consul-connect-injector.default,consul-connect-injector.default.svc,consul-connect-injector.default.svc...
{{- .Data.certificate -}}
{{- end -}}
vault.hashicorp.com/agent-inject-template-tls.key:
{{- with secret "connect-inject/issue/connect-inject-role" "common_name=consul-connect-injector"
"alt_names=consul-connect-injector,consul-connect-injector.default,consul-connect-injector.default.svc,consul-connect-injector.default.svc...
{{- .Data.private_key -}}
{{- end -}}
vault.hashicorp.com/role: connect-inject-role
vault.hashicorp.com/secret-volume-path-ca.crt: /vault/secrets/connect-injector/certs
vault.hashicorp.com/secret-volume-path-tls.crt: /vault/secrets/connect-injector/certs
vault.hashicorp.com/secret-volume-path-tls.key: /vault/secrets/connect-injector/certs
Status: Running
IP: 10.244.3.24
IPs:
IP: 10.244.3.24
Controlled By: ReplicaSet/consul-connect-injector-57b7c994b8
Init Containers:
vault-agent-init:
Container ID: containerd://15568da5619458fb3294a1aa2aa312e847e02996b591d367c655dbe93366ee4c
Image: hashicorp/vault:1.16.1
Image ID: docker.io/hashicorp/vault@sha256:5efe43c659d4dc53539a589a95d5ffe77fffb761b407766e0408b0b18c30cad8
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 19 Apr 2024 16:22:33 +0530
Finished: Fri, 19 Apr 2024 16:22:34 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
NAMESPACE: default (v1:metadata.namespace)
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_LOG_LEVEL: info
VAULT_LOG_FORMAT: standard
VAULT_CONFIG: eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJ....
Mounts:
/home/vault from home-init (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ftftl (ro)
/vault/secrets from vault-secrets (rw)
/vault/secrets/connect-injector/certs from vault-secrets-custom-0 (rw)
Containers:
sidecar-injector:
Container ID: containerd://72007fb998128808dc58f78effe8a9b9728a1b0e8500ad9b3a63eb24efce6985
Image: hashicorp/consul-k8s-control-plane:1.4.1
Image ID: docker.io/hashicorp/consul-k8s-control-plane@sha256:9bcbf07a652553ca17921f9f5b880f7ef774d23a475586d2bea53005aede626c
Port: 8080/TCP
Host Port: 0/TCP
SeccompProfile: RuntimeDefault
Command:
/bin/sh
-ec
exec consul-k8s-control-plane inject-connect \
-log-level=info \
-log-json=false \
-default-inject=false \
-consul-image="hashicorp/consul-enterprise:1.18.1-ent" \
-consul-dataplane-image="hashicorp/consul-dataplane:1.4.1" \
-consul-k8s-image="hashicorp/consul-k8s-control-plane:1.4.1" \
-release-name="consul" \
-release-namespace="default" \
-resource-prefix=consul \
-listen=:8080 \
-default-enable-transparent-proxy=true \
-enable-cni=false \
-transparent-proxy-default-overwrite-probes=true \
-enable-consul-dns=true \
-default-enable-metrics=false \
-enable-gateway-metrics=true \
-default-enable-metrics-merging=false \
-default-merged-metrics-port=20100 \
-default-prometheus-scrape-port=20200 \
-default-prometheus-scrape-path="/metrics" \
-allow-k8s-namespace="*" \
-enable-namespaces=true \
-consul-destination-namespace=default \
-enable-k8s-namespace-mirroring=true \
-tls-cert-dir=/vault/secrets/connect-injector/certs \
-enable-webhook-ca-update \
-default-envoy-proxy-concurrency=2 \
-default-enable-sidecar-proxy-lifecycle=true \
-default-enable-sidecar-proxy-lifecycle-shutdown-drain-listeners=true \
-default-sidecar-proxy-lifecycle-shutdown-grace-period-seconds=30 \
-default-sidecar-proxy-lifecycle-graceful-port=20600 \
-default-sidecar-proxy-lifecycle-graceful-shutdown-path="/graceful_shutdown" \
-default-sidecar-proxy-startup-failure-seconds=0 \
-default-sidecar-proxy-liveness-failure-seconds=0 \
-init-container-memory-limit=150Mi \
-init-container-memory-request=25Mi \
-init-container-cpu-request=50m \
-enable-auto-encrypt \
-enable-telemetry-collector=false \
State: Running
Started: Fri, 19 Apr 2024 16:22:35 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 50m
memory: 200Mi
Requests:
cpu: 50m
memory: 200Mi
Liveness: http-get http://:9445/readyz/ready delay=1s timeout=5s period=10s #success=1 #failure=2
Readiness: http-get http://:9445/readyz/ready delay=2s timeout=5s period=10s #success=1 #failure=2
Startup: http-get http://:9445/readyz/ready delay=30s timeout=5s period=2s #success=1 #failure=15
Environment:
NAMESPACE: default (v1:metadata.namespace)
POD_NAME: consul-connect-injector-57b7c994b8-jcl6c (v1:metadata.name)
CONSUL_ADDRESSES: consul-server.default.svc
CONSUL_GRPC_PORT: 8502
CONSUL_HTTP_PORT: 8501
CONSUL_DATACENTER: dc1
CONSUL_API_TIMEOUT: 5s
CONSUL_USE_TLS: true
CONSUL_CACERT_FILE: /vault/secrets/serverca.crt
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ftftl (ro)
/vault/secrets from vault-secrets (rw)
/vault/secrets/connect-injector/certs from vault-secrets-custom-0 (rw)
vault-agent:
Container ID: containerd://63521f67bb36326b28ed5d328aced056af3c11b8eeb6b3b6adfd01e73fc63ea6
Image: hashicorp/vault:1.16.1
Image ID: docker.io/hashicorp/vault@sha256:5efe43c659d4dc53539a589a95d5ffe77fffb761b407766e0408b0b18c30cad8
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
State: Running
Started: Fri, 19 Apr 2024 16:22:36 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
NAMESPACE: default (v1:metadata.namespace)
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_LOG_LEVEL: info
VAULT_LOG_FORMAT: standard
VAULT_CONFIG: eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGV....
Mounts:
/home/vault from home-sidecar (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ftftl (ro)
/vault/secrets from vault-secrets (rw)
/vault/secrets/connect-injector/certs from vault-secrets-custom-0 (rw)
...
user1@server-GJ9F6YC4M5 ~ % k exec -it consul-connect-injector-57b7c994b8-jcl6c -c sidecar-injector -- sh
/ $ cd /vault/secrets/connect-injector/certs
/vault/secrets/connect-injector/certs $ ls
ca.crt tls.crt tls.key
/vault/secrets/connect-injector/certs $
/vault/secrets/connect-injector/certs $
/vault/secrets/connect-injector/certs $ cat tls.crt
-----BEGIN CERTIFICATE-----
MIID5zCCAs+gAwIBAgIUUXGbaATAJ+q5IhgoO0c9w3xd/IQwDQYJKoZIhvcNAQEL
BQAwIjEgMB4GA1UEAxMXY29uc3VsLWNvbm5lY3QtaW5qZWN0b3IwHhcNMjQwNDE5
MTA1MjA2WhcNMjQwNTE5MTA1MjM2WjAiMSAwHgYDVQQDExdjb25zdWwtY29ubmVj
dC1pbmplY3RvcjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANghJJ94
vwFwMN/p8kebeGjcwWKYYUn2kpXmBKQ/K90uW5/2/8t7eKDW6B5knr+d5c1whwDV
5SwKP52TGaHDCGIrOdmG9e9HvLijzFJ6EzbrK3dTmZ3An+h/NOTYGBlZ7bPT6HGj
v3F+gq+mlR+dwNnsZIpAiTHgK7UrBghmzdR+OhY2/4YfXswTLsBPiLUjVz2ygkJU
V32JpP9c3wQy3ckUq2ZGz92FwrVuQrYjn4TqiHhfh/xDht85iN0gcYue0GtiJBRR
WwHtXMhvt2uU7D8EYQc8voUp/zvobFUSpr6yGA1mv/1qtsgHr/miWGq0PwXpaW5R
...
-----END CERTIFICATE-----/vault/secrets/connect-injector/certs $
/vault/secrets/connect-injector/certs $ openssl x509 -in tls.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
51:71:9b:68:04:c0:27:ea:b9:22:18:28:3b:47:3d:c3:7c:5d:fc:84
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = consul-connect-injector
Validity
Not Before: Apr 19 10:52:06 2024 GMT
Not After : May 19 10:52:36 2024 GMT
Subject: CN = consul-connect-injector
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:d8:21:24:9f:78:bf:01:70:30:df:e9:f2:47:9b:
78:68:dc:c1:62:98:61:49:f6:92:95:e6:04:a4:3f:
2b:dd:2e:5b:9f:f6:ff:cb:7b:78:a0:d6:e8:1e:64:
9e:bf:9d:e5:cd:70:87:00:d5:e5:2c:0a:3f:9d:93:
19:a1:c3:08:62:2b:39:d9:86:f5:ef:47:bc:b8:a3:
cc:52:7a:13:36:eb:2b:77:53:99:9d:c0:9f:e8:7f:
34:e4:d8:18:19:59:ed:b3:d3:e8:71:a3:bf:71:7e:
82:af:a6:95:1f:9d:c0:d9:ec:64:8a:40:89:31:e0:
2b:b5:2b:06:08:66:cd:d4:7e:3a:16:36:ff:86:1f:
5e:cc:13:2e:c0:4f:88:b5:23:57:3d:b2:82:42:54:
57:7d:89:a4:ff:5c:df:04:32:dd:c9:14:ab:66:46:
cf:dd:85:c2:b5:6e:42:b6:23:9f:84:ea:88:78:5f:
87:fc:43:86:df:39:88:dd:20:71:8b:9e:d0:6b:62:
24:14:51:5b:01:ed:5c:c8:6f:b7:6b:94:ec:3f:04:
61:07:3c:be:85:29:ff:3b:e8:6c:55:12:a6:be:b2:
18:0d:66:bf:fd:6a:b6:c8:07:af:f9:a2:58:6a:b4:
3f:05:e9:69:6e:51:eb:78:d9:ba:23:93:57:70:0a:
81:f5
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Key Agreement
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
C9:64:3E:0D:53:CD:EA:EE:EA:34:E3:B4:4F:A9:8B:B1:C7:52:54:77
X509v3 Authority Key Identifier:
79:6C:AF:F8:49:21:D6:03:29:DD:34:69:DB:69:61:EA:73:4E:C0:3D
X509v3 Subject Alternative Name:
DNS:consul-connect-injector, DNS:consul-connect-injector.default, DNS:consul-connect-injector.default.svc, DNS:consul-connect-injector.default.svc.cluster.local
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
8c:85:6e:f9:8c:98:89:e8:95:3c:b2:6f:76:c3:c1:1a:64:1b:
3a:9a:4c:25:9f:fe:87:c9:e4:5e:fb:fd:45:ef:29:aa:e2:ae:
5d:df:f0:ed:01:3d:1d:d5:24:5e:c3:c5:a1:2d:4a:cc:4e:ea:
95:0f:2c:b3:c1:da:69:3b:d2:06:4f:66:72:75:b6:86:66:89:
34:c3:e2:78:68:fa:9a:68:78:21:5d:5d:11:1e:32:b6:3d:2b:
cf:09:34:0e:44:64:e5:b6:32:9e:d6:c2:a7:ac:82:2f:06:03:
31:10:d2:90:f3:b5:5a:11:36:9e:b9:b3:c7:69:de:63:f7:7c:
df:ec:c2:81:1c:ac:e3:c6:67:62:70:4b:a1:90:d7:ec:64:72:
af:e5:ab:1d:80:54:1c:e7:8c:70:02:68:47:cb:8f:d9:3e:f1:
f9:c6:ab:34:37:1d:65:93:2a:75:25:f2:2d:c8:51:a9:a4:98:
4c:e8:8c:f1:c5:0d:a8:b2:55:7f:43:71:22:3b:bf:31:e5:50:
cc:c4:84:b7:66:26:fb:66:d2:d4:e0:6b:b4:7c:6e:9a:63:59:
21:44:5a:95:79:36:05:99:b8:87:77:73:f0:3b:fb:8a:96:a5:
4c:4c:fc:3e:1e:ad:b6:79:82:b3:2d:19:32:c6:c0:d9:d6:0f:
be:de:b5:00
Conclusion
This document explains how Vault can act as certificate provider for connect-inject and controller and what all required settings/configuration have to be done at the vault’s end.
Vault as webhook certificate manager ensures that a valid certificate is updated to the mutatingwebhookconfiguration of connect inject to ensure that Kubernetes can communicate with the service securely.
References
https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-sidecar#kubernetes-sidecar
https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-external-vault