Introduction
When Vault is used on a distribution of Linux utilizing systemd, the Vault operational logs by default are written to journald. The default journald setting for log storage is auto
and as a result, Vault operational logs are lost after a node is restarted. Using this guide, you can update the journald configuration and define log retention settings based on criteria such as time to retain logs or maximum allotment of space on disk to consume.
Expected Outcome
Once completed and a Vault node is restarted, the Vault operational logs should still be present.
Prerequisites
- As with any system level change, we recommend testing the process in a development or pre-production environment before making changes to production.
- Permissions required to update the
/etc/systemd/journald.conf
file. - Permissions required to create the
/var/log/journal
directory.
Permissions are normally owned by the root user, meaning su
or sudo
access will typically be the requirement to satisfy.
Use Case
As the Vault operational logs are usually required in order to troubleshoot any potential issues with Vault, it is helpful to ensure logs are accessible after a node restart.
Procedure
- Consider the appropriate strategy for log storage retention. Should retention settings be based on a pre-defined disk space allocation, time based value, or both?
- Consider starting with small values and adjusting based on the rate of log generation observed. After having an adequate sample size, make informed adjustments as necessary.
-
Validate on a node running Vault that you are able to view the operational logs from the recent past. Value of date included for demonstration purposes only:
[user@node1]~% date Mon May 30 15:46:45 AEST 2022 [user@node1]~% sudo journalctl -u vault --since=yesterday | head -- Logs begin at Thu 2022-05-19 02:22:17 AEST, end at Mon 2022-05-30 15:46:47 AEST. -- May 29 00:00:01 node1 vault[1302]: 2022-05-29T00:00:01.745+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[] May 29 00:00:06 node1 vault[1302]: 2022-05-29T00:00:06.746+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[] May 29 00:00:11 node1 vault[1302]: 2022-05-29T00:00:11.747+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[] May 29 00:00:16 node1 vault[1302]: 2022-05-29T00:00:16.746+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[]
-
[Optional] Restart the Vault node and repeat the same step as above to confirm logs are no longer available for viewing.
-
Verify the
/var/log/journal
directory exists and if not, create it. Ensure it is owned and writable by the root user. - Open the file
/etc/systemd/journald.conf
in your text editor of choice. - Change the value for
Storage
fromauto
toPersistent
. - Change the value for
SystemMaxUse
to a value of your choosing, i.e512M
or1G
. -
Change the value for
MaxRetentionSec
to a value of your choosing, i.e1d
or1month
or1year
. - Save the file and exit the text editor.
- Confirm that log files are being written to the folder
/var/log/journal
. - Restart the node and repeat the second step to validate logs are still visible from the previous day.
Additional Information
-
Journald configuration reference: https://www.freedesktop.org/software/systemd/man/journald.conf.html