The information contained in this article has been verified as up-to-date on the date of the original publication of the article. HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.
All information contained in this article is for general information purposes only. Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.
When a consul snapshot is being built, it uses the machine's notion of a temporary directory (
/tmp for Linux). This can become an issue when trying to save on VM costs by sizing the default
/tmp directory too low (i.e., 2GB with a 4GB RaftDB).
How Consul uses Go to specify its Raft snapshot temp directory
The default location can vary depending on the operating system, but typically it is
/tmp. You can get more detailed information on default locations in the Go documentation for os.TempDir.
If you need to change this location, you can do so by setting the
TMPDIRenvironment variable for the Consul server processes. Keep in mind that setting the environment variable for the CLI client attempting to perform a snapshot save will have no effect. It must be set in the context of the server process. If you're using
systemd to manage your Consul server processes, then adding
Environment=TMPDIR=/path/to/dirto your Consul unit file will work.
- System Level admin access, as this will be required when performing a rolling restart of the Consul Server agents
WARNING: This operation requires a rolling restart of Consul server agents, triggering a leadership switch when the existing leader is restarted. Based on your organizations policies, ensure that you have engaged the relavant change management process if any.
- Determine the OS's current $TMPDIR setting:
dirname $(mktemp -u -t tmp.XXXXXXXXXX)
- Determine Consul's
systemdunit environment settings (if applicable):
systemctl show consul | grep ^Environment
- If determined you need to change this setting, change the
$TMPDIRvalue at the Consul process level:
systemd config (/etc/systemd/system/consul.service):
Alternatively, you can update an EnvironmentFile that the consul.service daemon uses
echo 'TMPDIR=/home/ubuntu' >> /etc/consul.d/consul.env
- If applicable, perform daemon-reload as required for any
systemdunit file changes:
- Ensure Consul server quorum (N/2+1) is being maintained, and initiate a rolling restart of the Consul Server agents, starting with the followers first and the leader last. Use the service manager in use to initiate the restart of the agents.
The following example uses
- The following example uses
systemctl restart consul.service
- Verify proper Raft temporary snapshot directory usage by manually capturing a Consul snapshot, and (optionally) using a Linux watch to monitor for temporary snapshot creation and removal.
# consul-server-1 terminal session01 - initiates snapshot save
$ consul snapshot save consul_snapshot.snap
# consul-server-1 terminal session02 - monitors creation and removal of raft tmp file
$ watch -n 0.5 /home/ubuntu