How to Change Consul's Raft Snapshot Default /tmp Directory

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

When a consul snapshot is being built, it uses the machine's notion of a temporary directory ( /tmp for Linux). This can become an issue when trying to save on VM costs by sizing the default /tmp directory too low (i.e., 2GB with a 4GB RaftDB). 

Background

How Consul uses Go to specify its Raft snapshot temp directory

The default location can vary depending on the operating system, but typically it is/tmp. You can get more detailed information on default locations in the Go documentation for os.TempDir.

If you need to change this location, you can do so by setting theTMPDIRenvironment variable for the Consul server processes. Keep in mind that setting the environment variable for the CLI client attempting to perform a snapshot save will have no effect. It must be set in the context of the server process. If you're using systemd to manage your Consul server processes, then addingEnvironment=TMPDIR=/path/to/dirto your Consul unit file will work.

Prerequisites

• System Level admin access, as this will be required when performing a rolling restart of the Consul Server agents

WARNING: This operation requires a rolling restart of Consul server agents, triggering a leadership switch when the existing leader is restarted. Based on your organizations policies, ensure that you have engaged the relavant change management process if any.

Procedure

• Determine the OS's current $TMPDIR setting:

dirname $(mktemp -u -t tmp.XXXXXXXXXX)

• Determine Consul's systemd unit environment settings (if applicable):

systemctl show consul | grep ^Environment

• If determined you need to change this setting, change the $TMPDIR value at the Consul process level:

Example systemd config (/etc/systemd/system/consul.service):

[Service]
Environment=TMPDIR=/home/ubuntu

Alternatively, you can update an EnvironmentFile that the consul.service daemon uses

Systemd Unit:
[Service]
EnvironmentFile=-/etc/consul.d/consul.env

Shell:
echo 'TMPDIR=/home/ubuntu' >> /etc/consul.d/consul.env

• If applicable, perform daemon-reload as required for any systemd unit file changes:

systemctl daemon-reload

• Ensure Consul server quorum (N/2+1) is being maintained, and initiate a rolling restart of the Consul Server agents, starting with the followers first and the leader last. Use the service manager in use to initiate the restart of the agents.

• • The following example uses systemd

systemctl restart consul.service

• Verify proper Raft temporary snapshot directory usage by manually capturing a Consul snapshot, and (optionally) using a Linux watch to monitor for temporary snapshot creation and removal.

# consul-server-1 terminal session01 - initiates snapshot save

$ consul snapshot save consul_snapshot.snap

# consul-server-1 terminal session02 - monitors creation and removal of raft tmp file

$ watch -n 0.5 /home/ubuntu 

References:

• Consul & Raft Consensus Protocol
• Consul CLI Command: consul-snapshot-save