This KB article provides guidance on configuring the Service Definition file for the Consul Snapshot Agent to automate snapshots in Consul. It outlines the steps to add the systemctl file, apply and start the service, and verify service logs and created snapshots. By following these instructions, users can effectively manage Consul snapshots through automated processes, ensuring data integrity and operational efficiency.
Prerequisites:
- The agent subcommand described here is available only in Consul Enterprise version 0.7.1 and later. All other snapshot subcommands are available in the community edition of Consul.
- Consul snapshot agent configuration file.
- Please note that `systemd` is Linux-specific, so the procedures outlined in this article are applicable only to Linux systems.
Create Service definition file:
Below is a sample service definition file for the Snapshot Agent:
$ cat /etc/systemd/system/consul-snapshot.service
[Unit]
Description="Consul Snapshot Agent"
Documentation=https://www.consul.io/
Requires=network-online.target
After=consul.service
ConditionFileNotEmpty=/etc/consul.d/snapshot_agent.json
[Service]
Type=simple
User=consul
Group=consul
ExecStart=/usr/bin/consul snapshot agent -config-file=/etc/consul.d/snapshot_agent.json
KillMode=mixed
Restart=on-failure
RestartSec=10
StartLimitInterval=60
StartLimitBurst=6
LimitNOFILE=65536
[Install]
WantedBy=multi-user.targetYou can tune these settings to values that make sense for your workload.
For further details, please go through the following links.
sect-Managing_Services_with_systemd-Unit_File_Structure
systemd-automate-recovery
In above sample file "/etc/consul.d/snapshot_agent.json" is the snapshot agent configuration file. Please go through the link to know more about the snapshot agent configuration options.
Apply and Start newly added service:
You can run below commands to start the newly added service.
$ systemctl daemon-reload
# This command reloads systemd configuration files to ensure any recent changes take effect immediately.
$ systemctl status consul-snapshot
● consul-snapshot.service - "Consul Snapshot Agent"
Loaded: loaded (/etc/systemd/system/consul-snapshot.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: https://www.consul.io/
# This command displays the current status (active, inactive, or error) of a systemd unit (service, socket, device, etc.) along with relevant details and recent log entries.
$ systemctl start consul-snapshot
# This command initiates the execution of a systemd unit (service, socket, device, etc.), thereby starting the corresponding system component or application.
$ systemctl status consul-snapshot
● consul-snapshot.service - "Consul Snapshot Agent"
Loaded: loaded (/etc/systemd/system/consul-snapshot.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2024-06-28 09:14:37 UTC; 2s ago
Docs: https://www.consul.io/
Main PID: 61142 (consul)
Tasks: 6 (limit: 1121)
Memory: 17.9M
CPU: 69ms
CGroup: /system.slice/consul-snapshot.service
└─61142 /usr/bin/consul snapshot agent -config-file=/etc/consul.d/snapshot_agent.json
$ systemctl enable consul-snapshot
# This command configures Consul Snapshot as a systemd service to start automatically on system boot.
Cross Check the logs to make sure the things:
Jun 28 09:16:08 ip-172-31-82-84 consul[61142]: snapshot: Waiting to obtain leadership...
Jun 28 09:16:12 ip-172-31-82-84 consul[61142]: 2024-06-28T09:16:12.473Z [INFO] snapshot: Obtained leadership
Jun 28 09:16:12 ip-172-31-82-84 consul[61142]: snapshot: Obtained leadership
Jun 28 09:16:12 ip-172-31-82-84 consul[61142]: 2024-06-28T09:16:12.510Z [INFO] snapshot: Saved snapshot: id=1719566172506766168
Jun 28 09:16:12 ip-172-31-82-84 consul[61142]: snapshot: Saved snapshot: id=1719566172506766168
You can also Inspect the newly created snapshot by using "snapshot inspect" command.
$ consul snapshot inspect consul-1719566172506766168.snap
Additional Note
- It is always recommended to test any changes thoroughly in a non-production environment before deploying them to your production system.
- It is recommended obtaining the debug level log of the Consul snapshot agent when the issue begins and occurs would greatly assist us in analyzing simultaneous events and error messages. This will allow for a deeper understanding of the issue.