Overview

Ensuring the availability and integrity of your Consul cluster is critical for maintaining the consistency and reliability of your infrastructure. One effective strategy for achieving this is by leveraging Consul's snapshot feature, which allows you to back up and restore the state of your Consul cluster. By integrating Consul with Azure Blob Storage, you can securely store these snapshots in the cloud, benefiting from Azure's scalability, durability, and security.

This knowledge-based article provides a comprehensive guide on how to configure Consul to use Azure Blob Storage for storing snapshots. This document will walk you through the necessary steps to set up and authenticate Azure Blob Storage, generate a Shared Access Signature (SAS) token and Access Keys, and configure Consul to utilize these storage capabilities using these access keys.

Prerequisites

• Consul Enterprise v1.5.0 and later. Note: Try with latest consul versions to get benefit of recent features
• Azure CLI is installed.
• An Azure Storage account is created.
• Appropriate permissions to create and manage Azure resources.

Steps

Step 1: Install Azure CLI and Log In

• Install Azure CLI: Install Azure CLI using the below command -

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

For more details please refer to this official Install the Azure CLI on Linux url.

• Log in to Azure: use the below command from your Linux machine to login to Azure 

az login

Step 2: Create an Azure Storage Account and Container

• Create a Storage Account: Create a storage account where you want to push your Consul snapshot on the Azure blob. This storage account name will be used in the snapshot json configuration for Consul later in this article.

Syntax -

az storage account create --name YOUR_STORAGE_ACCOUNT_NAME --resource-group YOUR_RESOURCE_GROUP --location YOUR_LOCATION --sku Standard_LRS

Example - 

az storage account create --name consulsnapshotazure --resource-group aks-group --location "Central India" --sku Standard_LRS

• Create a Container in the Storage Account: Use the below command to create a container in your storage account. This will also be used in the Consul snapshot json configuration later.

Syntax -

az storage container create --name YOUR_CONTAINER_NAME --account-name YOUR_STORAGE_ACCOUNT_NAME

Example - 

az storage container create --name consul-snapshot --account-name consulsnapshotazure

Please note - Replace YOUR_STORAGE_ACCOUNT_NAME, YOUR_RESOURCE_GROUP, YOUR_LOCATION, and YOUR_CONTAINER_NAME with actual values from Azure portal. This document mentions the example part for the reference. 

Step 3: Generate a SAS Token

• Generate a SAS Token with the Necessary Permissions:

az storage account generate-sas --account-name YOUR_STORAGE_ACCOUNT_NAME --permissions acdlrw --resource-types sco --services b --expiry DATE_TIME --output tsv

Replace YOUR_STORAGE_ACCOUNT_NAME as per your step 2. Below is the reference -

Example - 

az storage account generate-sas --account-name consulsnapshotazure --permissions acdlrw --resource-types sco --services b --expiry 2024-12-31T23:59:00Z --output tsv

Please copy the output of the above command which is your generated SAS token. Ensure there are no line breaks or spaces in the token.

Note: This step is optional, as the current version of the Consul Snapshot Agent does not support the sas_token parameter within the snapshot JSON configuration or CLI commands. However, utilizing a SAS token for your Azure Storage account is recommended for enhanced security and access control.

Step 4: Fetch the Access Key from the Storage Account

• Fetch the access key for the storage account configured in Step 2. This access key will be used in the consul snapshot JSON configuration file. Use the below command - 

az storage account keys list --account-name YOUR_STORAGE_ACCOUNT_NAME --output tsv

Please note - Replace YOUR_STORAGE_ACCOUNT_NAME as per your step 2. Below is the example -

az storage account keys list --account-name consulsnapshotazure --output tsv

Step 5: Configure Consul Snapshot Configuration File

• Edit Your Consul snapshot JSON file (snapshot.json) to include Azure Blob Storage settings:

{
  "snapshot_agent": {
    "http_addr": "<consul_IP_ADDR>:8500",
    "token": "",
    "datacenter": "",
    "ca_file": "",
    "ca_path": "",
    "cert_file": "",
    "key_file": "",
    "license_path": "/etc/consul.d/license.hclic",
    "tls_server_name": "",
    "login": {
      "auth_method": "",
      "bearer_token": "",
      "bearer_token_file": "",
      "meta": {}
    },
    "log": {
      "level": "TRACE",
      "enable_syslog": false,
      "syslog_facility": "LOCAL0"
    },
    "snapshot": {
      "interval": "1h",
      "retain": 30,
      "stale": false,
      "service": "consul-snapshot",
      "deregister_after": "72h",
      "lock_key": "consul-snapshot/lock",
      "max_failures": 3,
      "local_scratch_path": ""
    },
    "backup_destinations": {
      "azure_blob_storage": [
        {
          "account_name": "YOUR_STORAGE_ACCOUNT_NAME",
          "account_key": "YOUR_ACCESS_KEY",
          "container_name": "YOUR_CONTAINER_NAME",
          "environment": "AzureCloud"
        }
      ]
    }
  }
}

Please note - 

1. YOUR_STORAGE_ACCOUNT_NAME will be replaced as per step 2.
2. YOUR_ACCESS_KEY will be replaced by the output retrieved in step 4.
3. YOUR_CONTAINER_NAME will be replaced as per step 2.
4. If "environment" is not working please check your Azure portal and replace it with the correct environment variable. You can also check the same in Azure CLI using the command "az cloud list --output table" and choose the correct value as per your setup.
5. If ACL and TLS are enabled please put the value of the related parameter as per your setup. 

Step 6: Start Consul Snapshot agent

• Use the below command to start the snapshot agent -

consul snapshot agent -config-file=/path/to/snapshot.json

Step 7: Validate the Snapshot Agent

• Check the logs for more details. Sample logs will be like below - 

==> Consul snapshot agent running!
             Version: 1.18.2+ent
          Datacenter: (default)
            Interval: "1h0m0s"
              Retain: 30
               Stale: false
       Local Scratch: /tmp
                Mode: daemon
             Service: "consul-snapshot"
    Deregister After: "72h0m0s"
            Lock Key: "consul-snapshot/lock"
        Max Failures: 3
    Snapshot Storage: Azure Blob Storage -> Environment: "AzureCloud" Account Name: "consulsnapshotazure" Container Name: "consul-snapshot"

==> Log data will now stream in as it occurs:

    2024-06-06T02:53:11.710+0530 [INFO]  snapshot: Waiting to obtain leadership...
    2024-06-06T02:53:11.746+0530 [INFO]  snapshot: Obtained leadership
    2024-06-06T02:53:11.763+0530 [DEBUG] snapshot: Taking a snapshot...
    2024-06-06T02:53:12.252+0530 [INFO]  azure: Upload of file to Azure Blob Storage successful: filename=consul-1717622591806899916.snap
    2024-06-06T02:53:12.252+0530 [INFO]  snapshot: Saved snapshot: id=1717622591806899916
    2024-06-06T02:53:12.372+0530 [DEBUG] snapshot: Rotated snapshots: number_deleted=0

• Check your Azure Blob Storage Container on Azure Portal to ensure the Snapshot file is uploaded.

• Use below Azure CLI command to validate the status of the consul snapshot -

az storage blob list --account-name YOUR_STORAGE_ACCOUNT_NAME --container-name YOUR_CONTAINER_NAME --query [].name --output tsv

Output from my lab -

user1@primary:/home/ubuntu# az storage blob list --account-name consulsnapshotazure --container-name consul-snapshot --query [].name --output tsv
consul-1717622591806899916.snap

Conclusion

By following the steps outlined above, you can successfully configure Consul to store snapshots in Azure Blob Storage which will provide a reliable backup solution for your Consul state data.

For further assistance, please refer to the Consul Snapshot Agent Documentation or contact your Azure support representative.