Vault as Consul Connect service mesh Certification Authority on Kubernetes

Avatar
Amir Akhavan
  • Updated

You can use Vault's PKI Secrets Engine to generate and renew dynamic X.509 certificates. Aside from being useful to secure Consul RPC and Raft communication, the PKI secrets engine can also act as a Certificate Authority (CA) for Consul Connect service mesh.

One of the benefits of using a service mesh is to have certificates for the service instances generated and rotated automatically and used to authenticate and secure service communication. Using Vault as CA for your service mesh allows you to use unified workflow for certificate generation and have a unified control over all Consul certificates' lifecycle.

In this article, you will learn how to configure your Consul datacenter to use Vault as the service mesh CA for new and existing data centers.

 

Configure Vault ACL

To interact with the PKI Secrets Engine endpoints, you have to generate a Vault token, giving the appropriate permissions to Consul to generate the certificates.

 

Create Vault policies

The following Vault policy allows Consul to create and manage the PKI paths in Vault. Consul is granted the ability to create the PKI mount points and given full control of the root and intermediate CA and to generate leaf certificates for Consul Connect service mesh clients.

The leaf certificates are created for each service instance and are used to verify the identity of the services and to secure communication across service instances.

# Consul Managed PKI Mounts
path "/sys/mounts" {
capabilities = [ "read" ]
}

path "/sys/mounts/connect_root" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/sys/mounts/connect_inter" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/connect_root/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/connect_inter/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}

Save the configuration in a file named vault-policy-connect-ca.hcl and create the policy with the Vault CLI.

$ vault policy write connect-ca vault-policy-connect-ca.hcl 
Success! Uploaded policy: connect-ca


Create Vault Token

Once the policy is created, create a Vault token to use for the integration.

$ vault token create -policy=connect-ca
Key Value
--- -----
token s.F2OdoyaVjP0jZ9IawYfAvQvc
token_accessor EwQDwlgPCLgqtXPGjzqNj7d1
token_duration 768h
token_renewable true
token_policies ["connect-ca" "default"]
identity_policies []
policies ["connect-ca" "default"]


Configure Consul service mesh CA

New Consul datacenter


Configure the service mesh CA using the server.extraConfig in the Consul configuration yaml file.

server:
enabled: true
extraConfig: |
{
"connect": {
"ca_provider": "vault",
"ca_config": {
"address": "http://172.31.58.83:8200",
"token": "s.F2OdoyaVjP0jZ9IawYfAvQvc",
"root_pki_path": "connect_root",
"intermediate_pki_path": "connect_inter"
}
}
}


 Deploy the Consul cluster via Helm using the updated Consul configuration above.

helm install consul hashicorp/consul -f consul.yaml


Verify Consul Connect CA configuration


$ consul connect ca get-config
{
"Provider": "vault",
"Config": {
"Address": "http://172.31.11.247:8200",
"IntermediateCertTTL": "8760h",
"IntermediatePKIPath": "connect_inter",
"LeafCertTTL": "72h",
"RootPKIPath": "connect_root",
"Token": "s.4fBS0tzPufo5s7VrjorFPoab"
},
"State": null,
"ForceWithoutCrossSigning": false,
"CreateIndex": 7,
"ModifyIndex": 7
}


Verify Certificate


curl http://172.31.21.80:8500/v1/agent/connect/ca/roots?pretty
{
"ActiveRootID": "c9:bf:62:44:db:66:5b:59:f5:25:6d:ba:af:77:95:19:b8:95:1f:70",
"TrustDomain": "7e85db9c-b390-ba6e-e171-651375d436fe.consul",
"Roots": [
{
"ID": "c9:bf:62:44:db:66:5b:59:f5:25:6d:ba:af:77:95:19:b8:95:1f:70",
"Name": "Vault CA Root Cert",
"SerialNumber": 3863032880979503790,
"SigningKeyID": "f1:5b:a3:3e:e7:60:b6:4c:55:e0:c4:9d:47:e3:5a:15:0e:90:67:5d",
"ExternalTrustDomain": "7e85db9c-b390-ba6e-e171-651375d436fe",
"NotBefore": "2021-11-23T21:07:12Z",
"NotAfter": "2021-12-25T21:07:42Z",
"RootCert": "-----BEGIN CERTIFICATE-----\nMIICLTCCAdKgAwIBAgIUCDt7qRGiLnNWXCAoNZw/8CJv6q4wCgYIKoZIzj0EAwIw\nLzEtMCsGA1UEAxMkcHJpLXd1dnpycjUudmF1bHQuY2EuN2U4NWRiOWMuY29uc3Vs\nMB4XDTIxMTEyMzIxMDcxMloXDTIxMTIyNTIxMDc0MlowLzEtMCsGA1UEAxMkcHJp\nLXd1dnpycjUudmF1bHQuY2EuN2U4NWRiOWMuY29uc3VsMFkwEwYHKoZIzj0CAQYI\nKoZIzj0DAQcDQgAEBngTxE3b4cQ165zVlNXiP/HT0O1qbky3Pm09HCiAKZENkd2c\nsDc+tdNx3tgInWPCvnl+j/gmCqNbrQ/flpuYeKOByzCByDAOBgNVHQ8BAf8EBAMC\nAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU197h3yz3CqKEod2lzja5Eola\nPecwHwYDVR0jBBgwFoAU197h3yz3CqKEod2lzja5EolaPecwZQYDVR0RBF4wXIIk\ncHJpLXd1dnpycjUudmF1bHQuY2EuN2U4NWRiOWMuY29uc3VshjRzcGlmZmU6Ly83\nZTg1ZGI5Yy1iMzkwLWJhNmUtZTE3MS02NTEzNzVkNDM2ZmUuY29uc3VsMAoGCCqG\nSM49BAMCA0kAMEYCIQDq93xC6+/eob7RBJ3Cl/YvLKHlc9/dmb181fY8vrrwHQIh\nAN0Byl6H5KYiS235xBm5c8Qq+rGFl19uufpQn8S7fj5H\n-----END CERTIFICATE-----\n",
"IntermediateCerts": null,
"Active": true,
"PrivateKeyType": "ec",
"PrivateKeyBits": 256,
"CreateIndex": 8,
"ModifyIndex": 8
}
]
}


Existing Consul datacenter

Verify existing configuration


$ consul connect ca get-config
{
"Provider": "consul",
"Config": {
"IntermediateCertTTL": "8760h",
"LeafCertTTL": "72h"
},
"State": null,
"ForceWithoutCrossSigning": false,
"CreateIndex": 5,
"ModifyIndex": 5
}


Change Consul Connect CA configuration

Once the datacenter is running, subsequent changes to the ca_provider configuration is ignored even if consul reload is ran or if the servers are restarted.To update any settings under this key, you must use Consul's update CA configuration API or the consul connect ca set-config command.First, create a file containing the new configuration for your datacenter CA in one of the server pods.


{
"Provider": "vault",
"Config": {
"Address": "http://172.31.48.157:8200",
"Token": "s.oGKuUi1PRAreouANIJbucpi3",
"RootPKIPath": "connect_root",
"IntermediatePKIPath": "connect_inter",
"LeafCertTTL": "72h",
"RotationPeriod": "2160h",
"IntermediateCertTTL": "8760h",
"PrivateKeyType": "rsa",
"PrivateKeyBits": 2048
},
"ForceWithoutCrossSigning": false
}


Next, apply the configuration to your Consul datacenter.

$ consul connect ca set-config -config-file config-connect-ca-provider-vault.json
Configuration updated!


The configuration change will trigger a root certificate rotation. An intermediate CA certificate is requested during rotation from the new root, which is then cross-signed by the old root and distributed alongside the newly generated leaf certificates.The cross-signing provides a chain of trust back to the old root certificate. This ensures new certificates are accepted also by proxies that have not completed the update to the new CA and avoids disruptions.After the configuration is changed, you should see the following line in your server logs.

[INFO] agent.server.connect: CA rotated to new root under provider: provider=vault


Verify Consul Connect CA configuration

Verify the CA settings by querying Consul for the CA configuration using the Consul CLI here.


$ consul connect ca get-config
{
"Provider": "vault",
"Config": {
"Address": "http://172.31.48.157:8200",
"IntermediateCertTTL": "8760h",
"IntermediatePKIPath": "connect_inter",
"LeafCertTTL": "72h",
"PrivateKeyBits": 2048,
"PrivateKeyType": "rsa",
"RootPKIPath": "connect_root",
"RotationPeriod": "2160h",
"Token": "s.oGKuUi1PRAreouANIJbucpi3"
},
"State": null,
"ForceWithoutCrossSigning": false,
"CreateIndex": 5,
"ModifyIndex": 201
}


Verify certificates


$ curl http://172.31.11.193:8500/v1/agent/connect/ca/roots?pretty
{
"ActiveRootID": "f5:97:32:d8:97:fb:1e:e1:9e:fc:13:d3:f8:a0:70:1e:d4:57:53:24",
"TrustDomain": "2017ac09-bb31-08f2-bd0d-f81948b0b476.consul",
"Roots": [
{
"ID": "e1:ca:6b:de:e6:0c:e2:8a:9a:3b:6f:4e:90:ae:dc:0c:32:19:40:c8",
"Name": "Consul CA Root Cert",
"SerialNumber": 7,
"SigningKeyID": "0a:fa:4c:11:d8:9a:4e:77:13:aa:aa:85:cc:c8:00:13:71:2e:96:8d:89:da:d4:9d:11:3b:9a:c3:18:23:3b:51",
"ExternalTrustDomain": "2017ac09-bb31-08f2-bd0d-f81948b0b476",
"NotBefore": "2021-11-23T21:26:29Z",
"NotAfter": "2031-11-23T21:26:29Z",
"RootCert": "-----BEGIN CERTIFICATE-----\nMIICEDCCAbWgAwIBAgIBBzAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktMWF0\nY25tcG4uY29uc3VsLmNhLjIwMTdhYzA5LmNvbnN1bDAeFw0yMTExMjMyMTI2Mjla\nFw0zMTExMjMyMTI2MjlaMDExLzAtBgNVBAMTJnByaS0xYXRjbm1wbi5jb25zdWwu\nY2EuMjAxN2FjMDkuY29uc3VsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEPX36\n+GdEX7Cwa84Gb5d7EqU2fMhQES5i5J4uUVR64aUCVU2tuxuL43GBmVLp6NHdeOOh\nWKuvilLpeY1mOV/zNqOBvTCBujAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUw\nAwEB/zApBgNVHQ4EIgQgCvpMEdiaTncTqqqFzMgAE3Eulo2J2tSdETuawxgjO1Ew\nKwYDVR0jBCQwIoAgCvpMEdiaTncTqqqFzMgAE3Eulo2J2tSdETuawxgjO1EwPwYD\nVR0RBDgwNoY0c3BpZmZlOi8vMjAxN2FjMDktYmIzMS0wOGYyLWJkMGQtZjgxOTQ4\nYjBiNDc2LmNvbnN1bDAKBggqhkjOPQQDAgNJADBGAiEAsPo7AQdq6mm7ABbNopTg\n6kl0uF7n+vaXp6yLq7serIkCIQDEYaNygyqK0VI8wvknaaQ/5UqWwbAmBUsen7Qf\npU8rLw==\n-----END CERTIFICATE-----\n",
"IntermediateCerts": null,
"Active": false,
"PrivateKeyType": "ec",
"PrivateKeyBits": 256,
"CreateIndex": 10,
"ModifyIndex": 201
},
{
"ID": "f5:97:32:d8:97:fb:1e:e1:9e:fc:13:d3:f8:a0:70:1e:d4:57:53:24",
"Name": "Vault CA Root Cert",
"SerialNumber": 1303275587784902772,
"SigningKeyID": "7d:5c:47:bf:4d:ef:f0:77:a5:3a:05:41:3f:f3:d5:0d:4c:8a:52:f7",
"ExternalTrustDomain": "2017ac09-bb31-08f2-bd0d-f81948b0b476",
"NotBefore": "2021-11-23T21:43:20Z",
"NotAfter": "2021-12-25T21:43:49Z",
"RootCert": "-----BEGIN CERTIFICATE-----\nMIIDuzCCAqOgAwIBAgIUA63iuiWhLQxOfzVmEhYqPfZbiHQwDQYJKoZIhvcNAQEL\nBQAwMDEuMCwGA1UEAxMlcHJpLTEzNDBla21pLnZhdWx0LmNhLjIwMTdhYzA5LmNv\nbnN1bDAeFw0yMTExMjMyMTQzMjBaFw0yMTEyMjUyMTQzNDlaMDAxLjAsBgNVBAMT\nJXByaS0xMzQwZWttaS52YXVsdC5jYS4yMDE3YWMwOS5jb25zdWwwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC5b2xX0MIJnrmWgyrAUZinFqb6xr5fc1Cv\nNzxx4doBpjMr7LZDQfGDFMNmlfMiUAksPSVkeSj8OHf0ZsmgjAeQMQpZlTDNhn8d\nVa87Wg8QbsbnFkPPRysvMQZ/XLWOwHgHj+BsaY4LJl5A4fw9nwALbWKUQd0VQKLq\nSCkOvrjQnPw4uyy44PWYNAfM3cpmlQzy0fmNSTchc7Vxu+M7hlPHqbimCw4vvWiO\ns1NGID0OpJ2sK+9oYJoywp0dtOI+CR9FDdmv88XKIF8MGJrMH7bD+gLukDiKJctj\nQXZqG/kGF70vsdKt4q5UruuGDwG5LNlIH8zSxkzkOtQBErWUwVD3AgMBAAGjgcww\ngckwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFH1c\nR79N7/B3pToFQT/z1Q1MilL3MB8GA1UdIwQYMBaAFH1cR79N7/B3pToFQT/z1Q1M\nilL3MGYGA1UdEQRfMF2CJXByaS0xMzQwZWttaS52YXVsdC5jYS4yMDE3YWMwOS5j\nb25zdWyGNHNwaWZmZTovLzIwMTdhYzA5LWJiMzEtMDhmMi1iZDBkLWY4MTk0OGIw\nYjQ3Ni5jb25zdWwwDQYJKoZIhvcNAQELBQADggEBADVao9SP5hdHbwqKCk9hIG+u\nwOhS0QbhkfsNgnhPl+bCXkSwy/WcjcC8SuVYLml/IcP/oZR9PTyMikFGvY0YjiTU\nIksOtSYjVhZ0/6C9eks2c4yc/DO7t+owvvX2c7AFxgzopkmQ5KUIfUxjmf+vrbEv\nxDjnazodHMJ/ERmIP6Ez8OiOwkamVQV3+t/k/tECD11qzRMSWgaBCpkEWqYhLz4U\nwWJUwu/J9x1FAgr04r36KKPvdwM90KNGeMM7HCSrcp0wPOclGyptglfiqfZ6Iwo7\nLZBErezwok1b32kPKNRw1vfy5cnoXyvDHlu2zMf2Qwy1rWMirVi30eT40HopYR4=\n-----END CERTIFICATE-----\n",
"IntermediateCerts": [
"-----BEGIN CERTIFICATE-----\nMIIC9TCCApqgAwIBAgIBCDAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktMWF0\nY25tcG4uY29uc3VsLmNhLjIwMTdhYzA5LmNvbnN1bDAeFw0yMTExMjMyMTQyNTBa\nFw0yMTExMzAyMTQyNTBaMDAxLjAsBgNVBAMTJXByaS0xMzQwZWttaS52YXVsdC5j\nYS4yMDE3YWMwOS5jb25zdWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB\nAQC5b2xX0MIJnrmWgyrAUZinFqb6xr5fc1CvNzxx4doBpjMr7LZDQfGDFMNmlfMi\nUAksPSVkeSj8OHf0ZsmgjAeQMQpZlTDNhn8dVa87Wg8QbsbnFkPPRysvMQZ/XLWO\nwHgHj+BsaY4LJl5A4fw9nwALbWKUQd0VQKLqSCkOvrjQnPw4uyy44PWYNAfM3cpm\nlQzy0fmNSTchc7Vxu+M7hlPHqbimCw4vvWiOs1NGID0OpJ2sK+9oYJoywp0dtOI+\nCR9FDdmv88XKIF8MGJrMH7bD+gLukDiKJctjQXZqG/kGF70vsdKt4q5UruuGDwG5\nLNlIH8zSxkzkOtQBErWUwVD3AgMBAAGjgdgwgdUwDgYDVR0PAQH/BAQDAgEGMA8G\nA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFH1cR79N7/B3pToFQT/z1Q1MilL3MCsG\nA1UdIwQkMCKAIAr6TBHYmk53E6qqhczIABNxLpaNidrUnRE7msMYIztRMGYGA1Ud\nEQRfMF2CJXByaS0xMzQwZWttaS52YXVsdC5jYS4yMDE3YWMwOS5jb25zdWyGNHNw\naWZmZTovLzIwMTdhYzA5LWJiMzEtMDhmMi1iZDBkLWY4MTk0OGIwYjQ3Ni5jb25z\ndWwwCgYIKoZIzj0EAwIDSQAwRgIhAPvx6L2nBn1LgAXhneYpOLXR1xO0tqjCghhr\nOIbDIRGJAiEAz0Us6AHiYSTcVbgMZ2wRWExG+lZ7KZQy0dPwitXaKhQ=\n-----END CERTIFICATE-----\n",
"-----BEGIN CERTIFICATE-----\nMIIDuzCCAqOgAwIBAgIUYnGG/A1+VQLx3SAiqGCWpPG1ohcwDQYJKoZIhvcNAQEL\nBQAwMDEuMCwGA1UEAxMlcHJpLTEzNDBla21pLnZhdWx0LmNhLjIwMTdhYzA5LmNv\nbnN1bDAeFw0yMTExMjMyMTQzMjBaFw0yMjExMjMyMTQzNTBaMDAxLjAsBgNVBAMT\nJXByaS05MGg3Ym8zci52YXVsdC5jYS4yMDE3YWMwOS5jb25zdWwwggEiMA0GCSqG\nSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDL+uupjY5xF/yd8Ytu5Baewrv4uU1azdCS\n7BlNatLIxsE5xJsitKa/I80Qm8R3dbLd7RDdczhK/h2f3mfZK47clpECOPHbsgNz\nfRB0GvxEwguihPuPs5lkWEBzSGqDj2IzXKxt2VtKjxbm3ozy90AyxHAI3PQpTCvU\nnUBsKgb1YfE9a3D1LbNBT1dKLpYCtKxVchTGH2cT03OaELRZpwWyQ7CGM6XrJ0Fb\nOpfOnv3jDACYMNFj1gQG8sgAFxWIprDPlCSZ9dndteIRvdZ9lbdn02CZHCvqtZsQ\ncL0pJi9W2eI/Xp6Y/nNUczbdJu//iYAjUk3iecI1ho6D9uIZC1THAgMBAAGjgcww\ngckwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFKE7\nkL0Pi1LdnB6g5lIlugsUrCvUMB8GA1UdIwQYMBaAFH1cR79N7/B3pToFQT/z1Q1M\nilL3MGYGA1UdEQRfMF2CJXByaS05MGg3Ym8zci52YXVsdC5jYS4yMDE3YWMwOS5j\nb25zdWyGNHNwaWZmZTovLzIwMTdhYzA5LWJiMzEtMDhmMi1iZDBkLWY4MTk0OGIw\nYjQ3Ni5jb25zdWwwDQYJKoZIhvcNAQELBQADggEBAAozh7hKpDrjdoCLPH/yTiCt\nYed37SIKcfQqDsyoOS0nezsn1WVyXGUT6tcwTCXrBiKBZ9aofEsfa3zIoUfSYK/q\nL/tbfxgrtsjAad09VpnOzLciiIy4DuKC7SaNr+mwGC+7cRWtlLshbZp6mzp8xMRA\n/Z/EHnNqErwq7Tad2p6FC7v8d6kxJQrregrR0JYd7I9HQmrKEpl3xdiUQLEMyh38\nN5mwAqmZxR3Ng3D9bTTe6uJVe+1mH8cKCt+CEDatDnl9zJNQODmgJsXv7gaoqd/x\npNIqHQcL8YIb0lOi9nqPON/jw64QUyR7B0jFiwd60B+MAPOpbt+2BvNhFQ5A5+s=\n-----END CERTIFICATE-----\n"
],
"Active": true,
"PrivateKeyType": "rsa",
"PrivateKeyBits": 2048,
"CreateIndex": 201,
"ModifyIndex": 201
}
]
}


Check leaf certificates for services

Use the /agent/connect/ca/leaf/<service> endpoint to retrieve the leaf certificates for the Consul Connect service mesh services.

 

Vault Namespaces

If you wish to create the PKI paths and their respective certificates in a specific Vault namespace, the namespace must be explicitly configured in Consul values used to install via Helm under server.extraEnvironmentVars.


New Consul datacenter

In addition to CA configuration, the extra environment variables is included in the configuration.

server:
enabled: true
extraConfig: |
{
"connect": {
"ca_provider": "vault",
"ca_config": {
"address": "http://172.31.48.157:8200",
"token": "s.z2i9MXV9SdOZIq3pV9LH7rcP",
"root_pki_path": "connect_root",
"intermediate_pki_path": "connect_inter"
}
}
}
extraEnvironmentVars:
VAULT_NAMESPACE: "consul/connect/"


Existing Consul datacenter

Prior to updating the CA configuration an existing Consul datacenter outlined above, Consul environment variables must be updated via Helm to include the Vault namespace.

server:
enabled: true
extraEnvironmentVars:
VAULT_NAMESPACE: "consul/connect/"

 

Additional Resources

 

 

Articles in this section

See more

Related articles