Introduction
Consul cluster installed on Kubernetes through Helm Chart has many underlying and unexplored concepts. One such concept is around use of TLS to ensure flawless & secure communications between Consul and Kubernetes components.
In this article, we will try to understand the request flow between consul-webhook-cert-manager and consul-connect-injector
Helm Chart
For the lab setup, you may set up a small K8s cluster. Ref. https://developer.hashicorp.com/consul/tutorials/kubernetes/kubernetes-kind
Once the K8s cluster is in place, then run a minimal consul cluster on it (through the helm chart) using the following "values.yaml" file.
values.yaml
global:
name: consul
connectInject:
enabled: true
server:
enabled: true
replicas: 3As soon as we deploy this file using the helm chart on K8s cluster, we could see the following components have been created.
NAME READY STATUS RESTARTS AGE
consul-connect-injector-58884f96b9-d5k82 1/1 Running 0 6d1h
consul-server-0 1/1 Running 0 6d1h
consul-server-1 1/1 Running 0 6d1h
consul-server-2 1/1 Running 0 6d1h
consul-webhook-cert-manager-75cf7ddf77-hh9mj 1/1 Running 0 6d1hwebhook-cert-manager logs
$ kubectl logs consul-webhook-cert-manager-75cf7ddf77-hh9mj
2024-04-11T06:12:36.345Z [INFO] Updated certificate bundle received for consul-connect-injector; Updating webhook certs.
2024-04-11T06:12:36.433Z [INFO] Creating Kubernetes secret with certificate: mutatingwebhookconfig=consul-connect-injector secret=consul-connect-inject-webhook-cert secretNS=default
2024-04-11T06:12:36.438Z [INFO] Updating webhook configuration: mutatingwebhookconfig=consul-connect-injector secret=consul-connect-inject-webhook-cert secretNS=defaultBy referring to the logs we can see that webhookCertManager does following three tasks:-
-
Generate a secret (consul-connect-inject-webhook-cert) with certificate in helm chart installed in namespace.
-
It will update connectInjector with certs from the above secret.
-
It will also update caBundle in mutatingWebhookConfiguration for consul-connect-injector
We will try to understand these pointers in much detail below:-
Generate a secret with certificates:-
Upon installation consul-webhook-cert-manager deployment creates one secret consul-connect-inject-webhook-cert which has tls.crt and tls.key. This certificate is valid for 24 hours by default.
user1@server-GJ9F6YC4M5 kind % kubectl get secrets
NAME TYPE DATA AGE
consul-connect-inject-webhook-cert kubernetes.io/tls 2 6d1h
sh.helm.release.v1.consul.v1 helm.sh/release.v1 1 6d1huser1@server-GJ9F6YC4M5 ~ % kubectl get secret consul-connect-inject-webhook-cert -o yaml
apiVersion: v1
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0F....
tls.key: LS0tLS1CRUdJTiBFQyBQUklWQVR....
kind: Secret
metadata:
creationTimestamp: "2024-04-11T06:12:36Z"
labels:
managed-by: consul-k8s
name: consul-connect-inject-webhook-cert
namespace: default
ownerReferences:
- apiVersion: apps/v1
kind: Deployment
name: consul-webhook-cert-manager
uid: ad80ce2c-2f0c-4826-bcc3-c9fe9c525c21
resourceVersion: "163921"
uid: 010c73fe-e7a8-48e0-997b-1f21b48fb9a7
type: kubernetes.io/tls
himanshusharma@himanshusharma-GJ9F6YC4M5 ~ %To review the certificate details use the below command.
$ echo "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0...." | base64 -d | openssl x509 -text -noout
Warning: Reading certificate from stdin since no -in or -new option is given
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
09:9f:0c:44:8f:fb:38:89:1f:b6:2a:09:71:87:fc:ad:78:82:a1:6a
Signature Algorithm: ecdsa-with-SHA256
Issuer: C=US, ST=CA, L=San Francisco, street=101 Second Street, postalCode=94105, O=HashiCorp Inc., CN=Consul Webhook Certificates CA
Validity
Not Before: Apr 16 07:43:18 2024 GMT
Not After : Apr 17 07:44:18 2024 GMT
Subject: CN=Consul Webhook Certificates Service
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:f4:fb:80:43:83:0f:ed:1c:0a:45:6b:fb:35:85:
33:e9:6e:88:d7:97:2c:f4:c4:19:34:c6:72:57:02:
7a:d1:46:b3:5f:9d:50:e1:65:d6:3c:0d:1d:20:0b:
af:2f:ba:e8:8a:d6:87:74:ee:f0:cb:48:a9:d9:16:
92:66:20:f5:a6
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
35:62:3A:33:35:3A:61:39:3A:37:32:3A:37:31:3A:31:64:3A:64:37:3A:62:33:3A:62:31:3A:64:65:3A:63:35:3A:31:62:3A:39:62:3A:39:34:3A:31:66:3A:35:36:3A:37:63:3A:30:33:3A:36:63:3A:37:31:3A:66:62:3A:39:64:3A:62:62:3A:35:31:3A:65:37:3A:32:66:3A:30:36:3A:30:66:3A:31:62:3A:65:34:3A:33:30:3A:32:31
X509v3 Subject Alternative Name:
DNS:consul-connect-injector, DNS:consul-connect-injector.default, DNS:consul-connect-injector.default.svc, DNS:consul-connect-injector.default.svc.cluster.local
Signature Algorithm: ecdsa-with-SHA256
Signature Value:
30:44:02:20:64:b0:70:03:4a:bb:58:51:f7:43:f2:6e:9c:0b:
77:b3:e5:b0:c0:30:f8:70:2b:af:2b:43:01:57:d3:7f:25:f8:
02:20:2c:ba:c2:50:63:19:c8:8b:2e:a2:be:f2:45:10:ae:48:
7c:1f:ce:3a:6d:2d:e4:5a:4f:b9:c7:e2:47:0e:c7:e1We could also see that reference for Subject Alternative Name for the above cert is received from configMap (Ref. https://github.com/hashicorp/consul-k8s/blob/main/charts/consul/templates/webhook-cert-manager-configmap.yaml ).
This configMap is passed as volume in webhook-cert-manager deployment. (Ref. https://github.com/hashicorp/consul-k8s/blob/8ac97bf0dc70576e01ade4337700c9f78ed54908/charts/consul/templates/webhook-cert-manager-deployment.yaml#L63-L71 )
Now, Webhook-Cert-Manager will keep on checking for the certificate expiring of the above cert (which is by default 24 hours - Ref. https://github.com/hashicorp/consul-k8s/blob/8ac97bf0dc70576e01ade4337700c9f78ed54908/control-plane/subcommand/webhook-cert-manager/command.go#L35-L38 ) and renew/update it within 10% of the time left for certificate expiration.
Update connectInjector with certs from secrets
The existing secret consul-connect-inject-webhook-cert will be used by consul-connect-inject (this secret is mounted as a volume to the pod). Using this certificate TLS/SSL connection between Controller (Admission Controller having MutatingWebhookConfig) -> Consul-Connect-Injector will be secured.
connect-injector-logs
$ kubectl logs consul-connect-injector-58884f96b9-d5k82
2024-04-11T06:13:15.836Z INFO controller-runtime.webhook.webhooks Starting webhook server
2024-04-11T06:13:15.837Z INFO Starting server {"kind": "health probe", "addr": "[::]:9445"}
2024-04-11T06:13:15.837Z INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:9444"}
2024-04-11T06:13:16.137Z INFO attempting to acquire leader lease default/consul-controller-lock...
2024-04-11T06:13:16.233Z INFO controller-runtime.certwatcher Updated current TLS certificate
2024-04-11T06:13:16.233Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 8080}
2024-04-11T06:13:16.234Z INFO controller-runtime.certwatcher Starting certificate watcher
2024-04-11T06:13:16.344Z INFO successfully acquired lease default/consul-controller-lock user1@server-GJ9F6YC4M5 ~ % kubectl get po consul-connect-injector-58884f96b9-d5k82 -o yaml
apiVersion: v1
kind: Pod
...
spec:
containers:
- command:
- /bin/sh
- -ec
- |
exec consul-k8s-control-plane inject-connect \
...
-tls-cert-dir=/etc/connect-injector/certs \
... \
...
volumeMounts:
- mountPath: /etc/connect-injector/certs
name: certs
readOnly: true
...
volumes:
- name: certs
secret:
defaultMode: 420
secretName: consul-connect-inject-webhook-cert
...
As soon as this certificate expires, webhookCertManager will renew the certificate, which will be fetched by connectInjector through continuous polling for any changes to be synced.
webhookCertManager logs when certs is renewed
kubectl logs consul-webhook-cert-manager-75cf7ddf77-hh9mj
...
2024-04-16T07:44:18.253Z [INFO] Updated certificate bundle received for consul-connect-injector; Updating webhook certs.
2024-04-16T07:44:18.343Z [INFO] Updating secret with new certificate: mutatingwebhookconfig=consul-connect-injector secret=consul-connect-inject-webhook-cert secretNS=default
2024-04-16T07:44:18.357Z [INFO] Updating webhook configuration with new CA: mutatingwebhookconfig=consul-connect-injector secret=consul-connect-inject-webhook-cert secretNS=defaultconnectInjector logs showing new certs being fetched
user1@server-GJ9F6YC4M5 kind % k logs consul-connect-injector-58884f96b9-d5k82
...
2024-04-16T07:44:36.444Z INFO controller-runtime.certwatcher Updated current TLS certificate
2024-04-16T07:44:36.445Z INFO controller-runtime.certwatcher Updated current TLS certificate
...
Update caBundle in mutatingWebhookConfiguration for connectInjector
WebhookCertManager is using certWatcher to continuously listen for a new MetaBundle for all webhooks and updates MutatingWebhooksConfigs and Secrets when a new Bundle is available. (Ref. https://github.com/hashicorp/consul-k8s/blob/8ac97bf0dc70576e01ade4337700c9f78ed54908/control-plane/subcommand/webhook-cert-manager/command.go#L202-L228 )
WebhookCertManager also uses reconcileCertificates to ensures the secret in the MetaBundle has the latest certificate from the MetaBundle and the caBundles on the MutatingWebhookConfiguration have the latest CA certificate from the MetaBundle. It updates them if they are outdated and exits early if they are up-to date. (Ref. https://github.com/hashicorp/consul-k8s/blob/8ac97bf0dc70576e01ade4337700c9f78ed54908/control-plane/subcommand/webhook-cert-manager/command.go#L230-L319 )
user1@server-GJ9F6YC4M5 ~ % k get mutatingwebhookconfiguration
NAME WEBHOOKS AGE
consul-connect-injector 13 5d6h
user1@server-GJ9F6YC4M5 ~ %user1@server-GJ9F6YC4M5 ~ % k describe mutatingwebhookconfiguration consul-connect-injector
Name: consul-connect-injector
Namespace:
Labels: app=consul
app.kubernetes.io/managed-by=Helm
chart=consul-helm
component=connect-injector
heritage=Helm
release=consul
Annotations: meta.helm.sh/release-name: consul
meta.helm.sh/release-namespace: default
API Version: admissionregistration.k8s.io/v1
Kind: MutatingWebhookConfiguration
Metadata:
Creation Timestamp: 2024-04-11T06:12:22Z
Generation: 2
Resource Version: 1245
UID: 8a065570-8ef9-4dca-bca7-1748bb103fca
Webhooks:
Admission Review Versions:
v1beta1
v1
Client Config:
Ca Bundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSU....
Service:
Name: consul-connect-injector
Namespace: default
Path: /mutate-v1alpha1-proxydefaults
Port: 443
Failure Policy: Fail
Match Policy: Equivalent
Name: mutate-proxydefaults.consul.hashicorp.com
....user1@server-GJ9F6YC4M5 ~ % echo "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tC...."
| base64 -d | openssl x509 -text -noout
Warning: Reading certificate from stdin since no -in or -new option is given
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
69:6a:b7:33:36:33:99:cd:5e:99:48:d5:3b:f3:6b:8b:3b:e5:c0:88
Signature Algorithm: ecdsa-with-SHA256
Issuer: C=US, ST=CA, L=San Francisco, street=101 Second Street, postalCode=94105, O=HashiCorp Inc., CN=Consul Webhook Certificates CA
Validity
Not Before: Apr 11 06:11:36 2024 GMT
Not After : Apr 9 06:12:36 2034 GMT
Subject: C=US, ST=CA, L=San Francisco, street=101 Second Street, postalCode=94105, O=HashiCorp Inc., CN=Consul Webhook Certificates CA
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:03:f0:5b:a7:bc:f2:69:f4:05:bd:6a:01:f3:94:
19:23:0a:ed:f4:48:bd:a6:82:7a:3f:68:a0:e8:79:
2a:48:fc:0c:d7:8b:aa:56:b1:19:19:e1:8b:a3:bb:
1e:5a:2d:59:a9:87:81:cb:96:2d:03:13:0a:56:83:
8f:db:64:ce:19
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Certificate Sign, CRL Sign
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
35:62:3A:33:35:3A:61:39:3A:37:32:3A:37:31:3A:31:64:3A:64:37:3A:62:33:3A:62:31:3A:64:65:3A:63:35:3A:31:62:3A:39:62:3A:39:34:3A:31:66:3A:35:36:3A:37:63:3A:30:33:3A:36:63:3A:37:31:3A:66:62:3A:39:64:3A:62:62:3A:35:31:3A:65:37:3A:32:66:3A:30:36:3A:30:66:3A:31:62:3A:65:34:3A:33:30:3A:32:31
X509v3 Authority Key Identifier:
35:62:3A:33:35:3A:61:39:3A:37:32:3A:37:31:3A:31:64:3A:64:37:3A:62:33:3A:62:31:3A:64:65:3A:63:35:3A:31:62:3A:39:62:3A:39:34:3A:31:66:3A:35:36:3A:37:63:3A:30:33:3A:36:63:3A:37:31:3A:66:62:3A:39:64:3A:62:62:3A:35:31:3A:65:37:3A:32:66:3A:30:36:3A:30:66:3A:31:62:3A:65:34:3A:33:30:3A:32:31
Signature Algorithm: ecdsa-with-SHA256
Signature Value:
30:45:02:20:28:cb:89:a9:e0:ee:01:40:b8:25:22:b9:52:78:
6d:d7:27:a6:6f:3e:8e:56:4b:1f:90:1d:c8:36:54:6e:5e:78:
02:21:00:d2:ef:bd:01:6f:12:cc:56:d6:1d:1a:b1:38:8a:c9:
5e:4a:1d:26:7e:89:5f:9e:5f:d9:e9:1f:7d:8c:be:8f:77
user1@server-GJ9F6YC4M5 ~ %
Conclusion:-
This document helps to understand how webhook-cert-manager ensures that a valid certificate is updated to the mutatingwebhookconfiguration of connect inject to ensure that Kubernetes can communicate with the service securely.
In the next guide, we will try to understand how to configure the Consul Helm chart to use Vault as CA provider to issue TLS certificates for Consul for connect inject webhook, instead of using above method of webhookCertManager.
References:-
https://developer.hashicorp.com/consul/docs/k8s/helm#webhookcertmanager
https://developer.hashicorp.com/consul/docs/k8s/helm#webhookcertmanager
https://github.com/hashicorp/consul-k8s/blob/main/control-plane/subcommand/inject-connect/command.go