Introduction
When Vault is used on a distribution of Linux utilising systemd the Vault operational logs are written to journald, however the default journald setting for log storage is `auto`, as a result Vault operational logs are lost after a node is restarted. Using this guide you can update the journald configuration and define log retention settings based on criteria such as time to retain logs or maximum allotment of space on disk to consume.
Expected Outcome
Once completed and a Vault node is restarted the Vault operational logs should still be present.
Prerequisites
- As with any system level change we recommend testing the process in a development or pre-production environment before making changes to production.
- Any permissions required to update the
/etc/systemd/journald.conf
file. - Any permissions required to create the
/var/log/journal
directory. - Both of these are usually owned by the root user, meaning
su
orsudo
access will typically be the requirement to satisfy.
Use Case
As the Vault operational logs are usually required in order to troubleshoot potential issues if the steps in this guide are implemented then the Vault operational logs will be available for viewing and analysis even if a node has been restarted.
Procedure
- Consider the appropriate strategy for log storage retention - should retention settings be based on a pre-defined disk space allocation, a time based value such as one month, or both? Consider starting with small values and adjusting based on the rate of log generation observed after a one week period and making informed adjustments if necessary or unsure.
-
Validate on a node running Vault that you are able to view the operational logs from yesterday. Value of date included for demonstration purposes only.
[user@node1]~% date Mon May 30 15:46:45 AEST 2022 [user@node1]~% sudo journalctl -u vault --since=yesterday | head -- Logs begin at Thu 2022-05-19 02:22:17 AEST, end at Mon 2022-05-30 15:46:47 AEST. -- May 29 00:00:01 node1 vault[1302]: 2022-05-29T00:00:01.745+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[] May 29 00:00:06 node1 vault[1302]: 2022-05-29T00:00:06.746+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[] May 29 00:00:11 node1 vault[1302]: 2022-05-29T00:00:11.747+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[] May 29 00:00:16 node1 vault[1302]: 2022-05-29T00:00:16.746+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=[]
-
[Optional] Restart the Vault node and repeat the same step as above to confirm logs are no longer available for viewing.
-
If it does not already exist, create the folder
/var/log/journal
and ensure it is owned and writeable by the root user. - Open the file
/etc/systemd/journald.conf
in your text editor of choice. - Change the value for
Storage
fromauto
toPersistent
- Change the value for
SystemMaxUse
to a value of your choosing, i.e512M
or1G
-
Change the value for
MaxRetentionSec
to a value of your choosing, i.e1d
or1month
or1year
- Save the file and exit the text editor.
- Confirm that log files are being written to the folder
/var/log/journal
. - Restart the node and repeat the second step to validate logs are still visible from the previous day.
Additional Information
-
Journald configuration reference: https://www.freedesktop.org/software/systemd/man/journald.conf.html