When Vault is used on a distribution of Linux utilising systemd the Vault operational logs are written to journald, however the default journald setting for log storage is `auto`, as a result Vault operational logs are lost after a node is restarted. Using this guide you can update the journald configuration and define log retention settings based on criteria such as time to retain logs or maximum allotment of space on disk to consume.
Once completed and a Vault node is restarted the Vault operational logs should still be present.
- As with any system level change we recommend testing the process in a development or pre-production environment before making changes to production.
- Any permissions required to update the
- Any permissions required to create the
- Both of these are usually owned by the root user, meaning
sudoaccess will typically be the requirement to satisfy.
As the Vault operational logs are usually required in order to troubleshoot potential issues if the steps in this guide are implemented then the Vault operational logs will be available for viewing and analysis even if a node has been restarted.
- Consider the appropriate strategy for log storage retention - should retention settings be based on a pre-defined disk space allocation, a time based value such as one month, or both? Consider starting with small values and adjusting based on the rate of log generation observed after a one week period and making informed adjustments if necessary or unsure.
Validate on a node running Vault that you are able to view the operational logs from yesterday. Value of date included for demonstration purposes only.
[user@node1]~% date Mon May 30 15:46:45 AEST 2022 [user@node1]~% sudo journalctl -u vault --since=yesterday | head -- Logs begin at Thu 2022-05-19 02:22:17 AEST, end at Mon 2022-05-30 15:46:47 AEST. -- May 29 00:00:01 node1 vault: 2022-05-29T00:00:01.745+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses= May 29 00:00:06 node1 vault: 2022-05-29T00:00:06.746+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses= May 29 00:00:11 node1 vault: 2022-05-29T00:00:11.747+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses= May 29 00:00:16 node1 vault: 2022-05-29T00:00:16.746+1000 [TRACE] core: replication: successful heartbeat: cluster_addresses=
[Optional] Restart the Vault node and repeat the same step as above to confirm logs are no longer available for viewing.
If it does not already exist, create the folder
/var/log/journaland ensure it is owned and writeable by the root user.
- Open the file
/etc/systemd/journald.confin your text editor of choice.
- Change the value for
- Change the value for
SystemMaxUseto a value of your choosing, i.e
Change the value for
MaxRetentionSecto a value of your choosing, i.e
- Save the file and exit the text editor.
- Confirm that log files are being written to the folder
- Restart the node and repeat the second step to validate logs are still visible from the previous day.
Journald configuration reference: https://www.freedesktop.org/software/systemd/man/journald.conf.html