Introduction
Consul Enterprise enables you to run the snapshot agent within your environment as a service (Systemd as an example) or scheduled through other means. Once running, the snapshot agent service operates as a highly available process that integrates with the snapshot API to automatically manage taking snapshots, backup rotation, and sending backup files offsite to Amazon S3 (or another S3-compatible endpoint), Google Cloud Storage, or Azure Blob Storage.
This capability provides an enterprise solution for backup and restoring the state of Consul servers within an environment in an automated manner. These snapshots are atomic and point-in-time. Consul datacenter backups include (but are not limited to):
- Key/Value Store Entries
- Service Catalog Registrations
- Prepared Queries
- Sessions
- Access Control Lists (ACLs)
- Namespaces
Use Case
This guide outlines the steps on how to configure the snapshot agent when operating Consul in a Kubernetes environment. Using Consul’s official Helm chart, the snapshot agents can be configured to set up and run snapshot agents within the Consul cluster. They are required to be co-located with Consul clients, so will inherit the clients’ nodeSelector, tolerations and affinity.
Procedure
Configure Consul Snapshot Agent
The following sample configuration can be used and modified accordingly to configure the Consul snapshot agent.
{
"snapshot_agent": {
"token": "",
"datacenter": "",
"ca_file": "",
"ca_path": "",
"cert_file": "",
"key_file": "",
"tls_server_name": "",
"log": {
"level": "INFO",
"enable_syslog": false,
"syslog_facility": "LOCAL0"
},
"snapshot": {
"interval": "1h",
"retain": 30,
"stale": false,
"service": "consul-snapshot",
"deregister_after": "72h",
"lock_key": "consul-snapshot/lock",
"max_failures": 3,
"local_scratch_path": ""
},
"local_storage": {
"path": "."
},
"aws_storage": {
"access_key_id": "",
"secret_access_key": "",
"s3_region": "",
"s3_bucket": "",
"s3_key_prefix": "consul-snapshot",
"s3_server_side_encryption": false,
"s3_static_snapshot_name": ""
},
"azure_blob_storage": {
"account_name": "",
"account_key": "",
"container_name": ""
},
"google_storage": {
"bucket": ""
}
}
}
Create Kuberentes Secret
Since Consul snapshot agent configuration contains sensitive information such as access key to various supported storages as well as ACL token, a Kubernetes secret containing the configuration file must be created using the following command:
kubectl create secret generic <SECRET NAME> --from-file=snapshot-agent=<PATH TO SNAPSHOT AGENT CONFIG>
Update Consul Helm Chart
Starting Consul version 1.14+ent, Consul snapshot-agent runs as a sidecar with Consul servers. See consul-k8s change-log and the respective PR.
This results in the following changes to Helm values:
- Move
client.snapshotAgent
values toserver.snapshotAgent
, with the exception of the following values:client.snaphostAgent.replicas
client.snaphostAgent.serviceAccount
- Remove
global.secretsBackend.vault.consulSnapshotAgentRole
value if applicable. You should now use theglobal.secretsBackend.vault.consulServerRole
for access to any Vault secrets.
With the secret containing the snapshot agent configuration in place, proceed with enabling the Consul snapshot agent and set the required replicas, secrets, and resources.
# select server or client based on the version of Consul in use.
server/client:
# snapshotAgent contains settings for setting up and running snapshot agents
# within the Consul clusters. They are required to be co-located with Consul
# clients, so will inherit the clients' nodeSelector, tolerations and affinity.
# This is an Enterprise feature only.
snapshotAgent:
enabled: true
# replicas determines how many snapshot agent pods are created
replicas: 2
# configSecret references a Kubernetes secret that should be manually created to
# contain the entire config to be used on the snapshot agent. This is the preferred
# method of configuration since there are usually storage credentials present.
# Snapshot agent config details:
# https://www.consul.io/docs/commands/snapshot/agent.html#config-file-options-
# To create a secret:
# https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-using-kubectl-create-secret
configSecret:
secretName: null
secretKey: null
# Resource settings for snapshot agent pods.
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
# Optional PEM-encoded CA certificate that will be added to the trusted system CAs.
# Useful if using an S3-compatible storage exposing a self-signed certificate.
# Example
# caCert: |
# -----BEGIN CERTIFICATE-----
# MIIC7jCCApSgAwIBAgIRAIq2zQEVexqxvtxP6J0bXAwwCgYIKoZIzj0EAwIwgbkx
# ...
caCert: null
Depnding on whether this is a new deployment or an existing one, apply the changes accordingly.
New deployment:
helm install <NAME OF DEPLOYMENT> hashicorp/consul -f <CONSUL VALUES.YAML>
Existing deployment:
helm upgrade <NAME OF DEPLOYMENT> hashicorp/consul -f <CONSUL VALUES.YAML>
Additional Information
Consul snapshot agent is an enterprise feature, please ensure that an enterprise image is referenced in Consul configuration.
If ACL is enabled in the cluster, the token with appropriate permissions for the snapshot agent must be included in the snapshot agent configuration. Additionally, if ACL is managed by Consul by setting manageSystemACLs
to true, the snapshot token should be obtained from Consul and included in the configuration in the subsequent Helm upgrade.
For more information on said permission, please review the documentation on Consul Frequently Used ACL Policies.
The following documents contain detailed information on Consul snapshot configuration.