Introduction
Consul uses the raft protocol for data replication between servers in the form of a replicated log. In order to constrain the size of the log, the raft protocol employs a periodic snapshot mechanism. This is performed automatically and without user intervention and only on servers.
The snapshots are located in the consul/data/raft/snapshot directory:
consul/data/raft/snapshots
├── 49-110398-1602903144301
│ ├── meta.json
│ └── state.bin
└── 49-110465-1602903150057
├── meta.json
└── state.bin
The directory naming follows the format TERM-INDEX-TIMESTAMP:
TERM - The current raft term.
INDEX - The index of the transaction log at the time of the snapshot
TIMESTAMP - The epoch timestamp of the snapshot
The corresponding Consul output log entries:
[INFO] agent.server.snapshot: creating new snapshot:path=/data/raft/snapshots/49-110398-1602903144301.tmp
[INFO] agent.server.snapshot: reaping snapshot: path=/data/raft/snapshots/49-110198-1602902768023
[INFO] agent.server.raft: compacting logs: from=100087 to=100160
[INFO] agent.server.raft: snapshot complete up to: index=110398
These snapshots allow Raft to capture the state at a point in time and then remove all prior log entries used to reach that state. Consul will keep the two most recent snapshots.
The state.bin file contains compressed state, while the meta.json file contains information about the environment at the time of the snapshot.
For example:
{
"Version": 1,
"ID": "49-110398-1602903144301",
"Index": 110398,
"Term": 49,
"Peers": "k64xMC4xMC4wLjM6ODMwMK4xMC4xMC4wLjQ6ODMwMK4xMC4xMC4wLjI6ODMwMA==",
"Configuration": {
"Servers": [
{
"Suffrage": 0,
"ID": "c8f4c9fc-4e26-7a5f-8c26-b173ad8f5b91",
"Address": "10.10.0.2:8300"
}
]
},
"ConfigurationIndex": 106326,
"Size": 15033,
"CRC": "4coef92etbE="
}
49-110398-1602903144301/meta.json
Examining the corresponding Consul output log
2020-10-17T02:52:24.301Z [INFO] agent.server.fsm: snapshot created: duration=77.048µs
2020-10-17T02:52:24.301Z [INFO] agent.server.raft: starting snapshot up to: index=110398
2020-10-17T02:52:24.301Z [INFO] agent.server.snapshot: creating new snapshot: path=/data/raft/snapshots/49-110398-1602903144301.tmp
2020-10-17T02:52:24.435Z [INFO] agent.server.snapshot: reaping snapshot: path=/data/raft/snapshots/49-110198-1602902768023
2020-10-17T02:52:24.436Z [INFO] agent.server.raft: compacting logs: from=100087 to=100160
2020-10-17T02:52:24.467Z [INFO] agent.server.raft: snapshot complete up to: index=110398
Detailed:
agent.server.fsm: snapshot created
The snapshot has been created in memory.
agent.server.raft: starting snapshot up to: index=110398
agent.server.snapshot: creating new snapshot: path=/data/raft/snapshots/49-110398-1602903144301.tmp
agent.server.snapshot: reaping snapshot: path=/data/raft/snapshots/49-110198-1602902768023
The snapshot up to index of 110398 was recorded to disc as was recorded to disk as "49-110398-1602903144301.tmp" and the oldest snapshot "49-110198-1602902768023" is destroyed. The ".tmp" extension is removed once the write and removal is successful.
agent.server.raft: compacting logs: from=100087 to=100160
agent.server.raft: snapshot complete up to: index=110398
Log entries 100087 to 100160 are archived, the current log contains entries from 100161 through 110398. This completes the snapshot.
During the next raft_snapshot_interval that the raft_snapshot_threshold has been met, the process is repeated and snapshot 49-110465-1602903150057 is recorded.
2020-10-17T02:52:30.057Z [INFO] agent.server.fsm: snapshot created: duration=62.491µs
2020-10-17T02:52:30.057Z [INFO] agent.server.raft: starting snapshot up to: index=110465
2020-10-17T02:52:30.057Z [INFO] agent.server.snapshot: creating new snapshot: path=/data/raft/snapshots/49-110465-1602903150057.tmp
2020-10-17T02:52:30.194Z [INFO] agent.server.snapshot: reaping snapshot: path=/data/raft/snapshots/49-110324-1602903137787
2020-10-17T02:52:30.194Z [INFO] agent.server.raft: compacting logs: from=100161 to=100227
2020-10-17T02:52:30.226Z [INFO] agent.server.raft: snapshot complete up to: index=110465
Tuning
Under normal operations, the snapshot process should not require any manual tuning. However, consul allows for limited configuration of the raft snapshot behavior through the use of the raft_snapshot_threshold and raft_snapshot_interval settings.
- raft_snapshot_threshold sets the minimum amount of change required before a snapshot is taken.
- raft_snapshot_interval sets how often this is checked.
Additionally, raft_trailing_logs controls the number of log entries left in the log store after a snapshot has been made. Under normal conditions, this should not need adjusting. Please see https://github.com/hashicorp/consul/pull/6186 for additional details.
Additional Information
https://www.consul.io/docs/agent/options.html#_raft_snapshot_threshold
https://www.consul.io/docs/agent/options.html#_raft_snapshot_interval
https://www.consul.io/docs/agent/options.html#raft_trailing_logs
https://www.consul.io/docs/architecture/consensus
https://en.wikipedia.org/wiki/Raft_(algorithm)