Consul Snapshot Agent: Capturing Snapshots from Local and Remote DCs – HashiCorp Help Center

Introduction

This guide details the configuration of the Consul snapshot agent to efficiently capture snapshots from both local and remote data centers (DCs) using a single node. By leveraging distinct configuration files, this method ensures isolated and robust snapshot management across your Consul environments.

Expected Outcome

Upon successful completion of this procedure, you will be able to:

Capture Consul snapshots from multiple data centers (e.g., dc1 and dc2) from a single, centralized node.
Schedule and automate recurring snapshots for continuous data protection.
Store snapshots locally with clear isolation between DC-specific configurations, preventing conflicts and improving organization.

Prerequisites

To successfully implement this solution, ensure the following are in place:

HashiCorp Consul Enterprise Edition: This feature is specifically available in the Enterprise version of Consul.
Access to Consul Agents: You must have appropriate network access to the Consul agents in all participating data centers.
Required ACL Token: If Access Control Lists (ACLs) are enabled in your Consul environment, a valid ACL token with snapshot read permissions is essential.
Proper TLS Configuration: If Transport Layer Security (TLS) is enabled for secure communication, ensure your TLS certificates and configurations are correctly set up on the snapshot agent node.
File System Access: The user running the snapshot agent must have read and write permissions to create and modify configuration files and store snapshots in the specified local paths.

Use Case

This centralized snapshot setup is particularly beneficial in scenarios where you need to:

Automate backup of Consul state across multiple data centers: Ensure consistent and timely backups without manual intervention for each DC.
Centralize snapshot management on a single server: Simplify operational oversight and reduce the number of machines dedicated to backup tasks.
Reduce operational overhead while managing multiple Consul environments: Consolidate backup procedures and minimize the administrative burden of protecting your Consul data.

Procedure

Step 1: Verify WAN Federation

Before configuring the snapshot agents, it is crucial to confirm that your chosen snapshot node can communicate with both the local and remote data centers via WAN federation. Use the consul members -wan command to verify connectivity:

consul members -wan
Node          Address         Status  Type    Build       Protocol  DC   Partition  Segment
server01.dc1  127.0.0.1:8302  alive   server  1.19.1+ent  2         dc1  default    <all>  <<<<<<<<<<<< local DC
server02.dc2  127.0.0.2:8302  alive   server  1.19.1+ent  2         dc2  default    <al    <<<<<<<<<<<< remote DC

This output confirms that server01.dc1 (our local DC in this example) and server02.dc2 (the remote DC) are both visible and alive over the WAN.

Step 2: Configure Snapshot for Remote DC (dc1)

Create a dedicated configuration file, for instance, snapshot_dc1.json, for the remote data center (dc1 in this example). This file will define the parameters for capturing snapshots from dc1.

{
  "snapshot_agent": {
    "http_addr": "127.0.0.1:8500",
    "token": "YOUR_ACL_TOKEN_HERE", // Replace with your actual ACL token if ACLs are enabled
    "datacenter": "dc1",
    "snapshot": {
      "interval": "5m",
      "max_snapshots": 10 // Retain the last 10 snapshots
    },
    "local_path": "/tmp/consul/snapshots/dc1" // Dedicated path for dc1 snapshots
  }
}

Key Configuration Details:

http_addr: The address of the Consul agent in dc1 that the snapshot agent will connect to.
token: Your Consul ACL token with appropriate permissions to read snapshot data.
datacenter: Specifies the target data center (dc1).
interval: How frequently snapshots are taken (e.g., "5m" for every 5 minutes).
max_snapshots: The number of recent snapshots to retain locally.
local_path: The directory where snapshots for dc1 will be stored. This should be a dedicated path.

Now, run the Consul snapshot agent using this configuration file:

consul snapshot agent -config-file=snapshot1.json

Example output:

Datacenter: "dc1"
Snapshot Storage: Local -> Path: "/tmp/consul/snapshots/dc1"
Mode: daemon

Step 3: Configure Snapshot for Local DC (dc2)

Similarly, create a separate configuration file, such as snapshot_dc2.json, for the local data center (dc2 in this example).

{
  "snapshot_agent": {
    "http_addr": "127.0.0.2:8500", // Assuming a different local IP for the dc2 agent
    "token": "YOUR_ACL_TOKEN_HERE", // Replace with your actual ACL token if ACLs are enabled
    "datacenter": "dc2",
    "snapshot": {
      "interval": "5m",
      "max_snapshots": 10
    },
    "local_path": "/tmp/consul/snapshots/dc2" // Dedicated path for dc2 snapshots
  }
}

Important: Ensure the http_addr points to the correct Consul agent for dc2, and local_path is distinct from the one used for dc1 to maintain isolation.

Run the Consul snapshot agent for the local DC:

consul snapshot agent -config-file=/etc/consul.d/snapshot_dc2.json

Example Output:

Datacenter: "dc2"
Snapshot Storage: Local -> Path: "/var/lib/consul/snapshots/dc2"
Mode: daemon

Step 4: Validate Snapshot Files

After allowing sufficient time for the configured interval to pass, verify that snapshots are being generated and stored correctly in their respective local_path directories.

For dc1 snapshots:

ls -l /var/lib/consul/snapshots/dc1/

You should see files similar to consul-snapshot-dc1-1678886400.snap.

For dc2 snapshots:

ls -l /var/lib/consul/snapshots/dc2/

You should see files similar to consul-snapshot-dc2-1678886400.snap.

Important Considerations:

Error Checking: Regularly check the logs of the consul snapshot agent processes for any errors or warnings related to snapshot creation or storage.
Monitoring: Implement monitoring for the snapshot directories to ensure disk space availability and the continuous generation of snapshots.
Backup Strategy: These local snapshots are a first line of defense. Integrate them into a broader backup strategy that includes off-site storage and disaster recovery plans.

Additional Information