The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

Consul provides mechanisms to back up cluster data using the consul snapshot command or the Consul Snapshot Agent. These tools create Raft snapshots, which are essential for cluster recovery and consistency. Each snapshot is packaged into a .snap file and typically stored in a designated directory.

Additionally, Consul automatically creates periodic Raft snapshots to ensure data safety. More details on Consul's consensus mechanism can be found in the Consul Consensus Documentation.

Problem

Accumulation of Temporary (.tmp) Files

Occasionally, when the consul snapshot command or Snapshot Agent fails to complete successfully, temporary directories (.tmp) may be left behind in the /opt/consul/data/raft/snapshots directory. These .tmp directories are not automatically cleaned up by Consul.

The /opt/consul/data/raft/snapshots directory usually contains two subdirectories representing the two most recent Raft snapshots. Over time, the accumulation of .tmp directories can cause the data directory to fill up, potentially impacting Consul’s functionality.

Cause

The issue arises when:

• The snapshot creation process is interrupted or fails.
• Temporary .tmp directories are generated but not removed.

Solution

To address this issue:

1. Manually Clean Up .tmp Directories

   • Navigate to the /opt/consul/data/raft/snapshots directory.
   • Identify and delete all .tmp directories except the two most recent snapshots.
   • Example command:

     cd /opt/consul/data/raft/snapshots
     ls -t | grep '\.tmp' | tail -n +3 | xargs rm -rf
     

     This command ensures that only older .tmp directories are removed, leaving the two most recent ones intact.

2. Leverage Consul Snapshot Agent Use the Consul Snapshot Agent for automated snapshot management and retention policies. Below are examples of HCL configurations for different storage options.

Local Storage Example

Store snapshots locally on the file system:

snapshot_agent {
  snapshot {
    interval            = "1h"
    retain              = 30
    local_scratch_path  = "/path/to/scratch"
  }

  backup_destinations {
    local_storage = [
      {
        path = "/backup/consul/snapshots"
      }
    ]
  }
}

AWS Storage Example

Store snapshots in an Amazon S3 bucket:

snapshot_agent {
  snapshot {
    interval            = "1h"
    retain              = 30
  }

  backup_destinations {
    aws_storage = [
      {
        access_key_id     = "your-access-key-id"
        secret_access_key = "your-secret-access-key"
        s3_region         = "your-region"
        s3_bucket         = "your-s3-bucket-name"
        s3_key_prefix     = "consul-snapshot"
      }
    ]
  }
}

Azure Blob Storage Example

Store snapshots in an Azure Blob Storage container:

snapshot_agent {
  snapshot {
    interval            = "1h"
    retain              = 30
  }

  backup_destinations {
    azure_blob_storage = [
      {
        account_name   = "your-account-name"
        account_key    = "your-account-key"
        container_name = "your-container-name"
      }
    ]
  }
}

Google Cloud Storage Example

Store snapshots in a Google Cloud Storage bucket:

snapshot_agent {
  snapshot {
    interval            = "1h"
    retain              = 30
  }

  backup_destinations {
    google_storage = [
      {
        bucket = "your-bucket-name"
      }
    ]
  }
}

Additional Recommendations

• Regularly monitor disk usage in the snapshot directory to prevent storage issues.
• Configure retention policies to automatically clean up older snapshots and avoid manual intervention.
• Consider offloading snapshots to remote storage (e.g., AWS S3, Azure Blob Storage, or Google Cloud Storage) for better scalability and redundancy.

By implementing these configurations and best practices, you can ensure your Consul snapshot directory remains clean, organized, and functional, minimizing the risk of disruptions caused by excessive .tmp files.

Reference Links

These references provide all the necessary resources to understand, diagnose, and resolve issues related to the Consul snapshot directory and .tmp files.

• Consul Snapshot Command Documentation
  Detailed documentation on using the consul snapshot command for managing backups and restores

• Consul Snapshot Agent Documentation
  Comprehensive guide on setting up and using the Consul Snapshot Agent, including configuration options and best practices

• Consul Consensus and Raft Protocol
  An in-depth explanation of Consul’s consensus algorithm and Raft protocol used for consistency and state replication

• Consul Configuration Options
  Official documentation for all configurable options available in Consul, including backup and snapshot settings

• Consul Data Directory Management
  Best practices for managing the Consul data directory, including storage location recommendations

• AWS S3 Storage Documentation
  Instructions for setting up and managing AWS S3 buckets, which can be used as a remote snapshot destination

• Azure Blob Storage Documentation
  Guide for configuring and using Azure Blob Storage for secure, scalable storage

• Google Cloud Storage Documentation
  Resources for setting up and managing Google Cloud Storage buckets

• Consul Troubleshooting Guide
  Common issues and troubleshooting steps for resolving operational problems with Consul

• Consul GitHub Repository
  Access to the source code, issues, and updates for Consul, including contributions from the community

• HashiCorp Learn Platform
  A learning resource with step-by-step tutorials on Consul features and integrations