This KB article provides guidance on configuring the Service Definition file for the Consul Snapshot Agent to automate snapshots in Consul. It outlines the steps to add the systemctl file, apply and start the service, and verify service logs and created snapshots. By following these instructions, users can effectively manage Consul snapshots through automated processes, ensuring data integrity and operational efficiency.
Prerequisites:
- The agent subcommand described here is available only in Consul Enterprise version 0.7.1 and later. All other snapshot subcommands are available in the community edition of Consul.
- Consul snapshot agent configuration file.
- Please note that `systemd` is Linux-specific, so the procedures outlined in this article are applicable only to Linux systems.
Create Service definition file:
Below is a sample service definition file for the Snapshot Agent:
$ cat /etc/systemd/system/consul-snapshot.service
[Unit]
Description="Consul Snapshot Agent"
Documentation=https://www.consul.io/
Requires=network-online.target
After=consul.service
ConditionFileNotEmpty=/etc/consul.d/snapshot_agent.json
[Service]
Type=simple
User=consul
Group=consul
ExecStart=/usr/bin/consul snapshot agent -config-file=/etc/consul.d/snapshot_agent.json
KillMode=mixed
Restart=on-failure
RestartSec=10
StartLimitInterval=60
StartLimitBurst=6
LimitNOFILE=65536
[Install]
WantedBy=multi-user.targetYou can tune these settings to values that make sense for your workload.
For further details, please go through the following links.
sect-Managing_Services_with_systemd-Unit_File_Structure
systemd-automate-recovery
In above sample file "/etc/consul.d/snapshot_agent.json" is the snapshot agent configuration file. Please go through the link to know more about the snapshot agent configuration options.
Reload, Enable and Start newly added service:
You can run below commands to start the newly added service.
$ systemctl daemon-reload
# This command reloads systemd configuration files to ensure any recent changes take effect immediately.
$ systemctl status consul-snapshot
● consul-snapshot.service - "Consul Snapshot Agent"
Loaded: loaded (/etc/systemd/system/consul-snapshot.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: https://www.consul.io/
# This command displays the current status (active, inactive, or error) of a systemd unit (service, socket, device, etc.) along with relevant details and recent log entries.
$ systemctl enable consul-snapshot
# This command configures Consul Snapshot as a systemd service to start automatically on system boot.
$ systemctl start consul-snapshot
# This command initiates the execution of a systemd unit (service, socket, device, etc.), thereby starting the corresponding system component or application.
$ systemctl status consul-snapshot
● consul-snapshot.service - "Consul Snapshot Agent"
Loaded: loaded (/etc/systemd/system/consul-snapshot.service; enabled; preset: disabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: active (running) since Sat 2025-01-04 01:09:54 UTC; 24s ago
Docs: https://www.consul.io/
Main PID: 167159 (consul)
Tasks: 7 (limit: 2207)
Memory: 17.8M (peak: 18.0M)
CPU: 31ms
CGroup: /system.slice/consul-snapshot.service
└─167159 /usr/bin/consul snapshot agent -config-file=/etc/consul.d/snapshot_agent.json
Jan 04 01:09:54 lima-kbsnapshot-srv-01 consul[167159]: Mode: daemon
Jan 04 01:09:54 lima-kbsnapshot-srv-01 consul[167159]: Service: "consul-snapshot"
Jan 04 01:09:54 lima-kbsnapshot-srv-01 consul[167159]: Deregister After: "72h0m0s"
Jan 04 01:09:54 lima-kbsnapshot-srv-01 consul[167159]: Lock Key: "consul-snapshot/lock"
Jan 04 01:09:54 lima-kbsnapshot-srv-01 consul[167159]: Max Failures: 3
Jan 04 01:09:54 lima-kbsnapshot-srv-01 consul[167159]: Snapshot Storage: Local -> Path: "/opt/consul/snapshots"
Jun 28 09:16:08 ip-172-31-82-84 consul[61142]: snapshot: Waiting to obtain leadership...
Jun 28 09:16:12 ip-172-31-82-84 consul[61142]: 2024-06-28T09:16:12.473Z [INFO] snapshot: Obtained leadership
Jun 28 09:16:12 ip-172-31-82-84 consul[61142]: snapshot: Obtained leadership
Jun 28 09:16:12 ip-172-31-82-84 consul[61142]: 2024-06-28T09:16:12.510Z [INFO] snapshot: Saved snapshot: id=1719566172506766168
Jun 28 09:16:12 ip-172-31-82-84 consul[61142]: snapshot: Saved snapshot: id=1719566172506766168
You can also Inspect the newly created snapshot by using "snapshot inspect" command.
$ consul snapshot inspect consul-1719566172506766168.snap
Additional Note
- It is always recommended to test any changes thoroughly in a non-production environment before deploying them to your production system.
- It is recommended obtaining the debug level log of the Consul snapshot agent when the issue begins and occurs would greatly assist us in analyzing simultaneous events and error messages. This will allow for a deeper understanding of the issue.